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INTERPRETIVE SUMMARY 
OF THE EVALUATION OF THE 
INSTRUCTIONAL MATERIALS 
DEVELOPMENT PROGRAM 

WestEd and Abt Associates Inc. conducted an 
evaluation of the National Science Foundation’s (NSF’s) 
Instructional Materials Development (IMD) program that 
focused on issues related to the development, dissemina- 
tion, adoption, implementation, and impact of materials 
created with NSF support. An Expert Panel comprising 
scientists; mathematicians; science, mathematics, and 
technology educators; educational administrators; and 
teachers provided input into the study (see Appendix 1 for 
the members of the Expert Panel). At the conclusion of the 
study, the Panel met to develop an interpretive summary of 
the evaluation, which is reflected in this document. The 
summary offers insight and advice to NSF, future developers 
andpublishers, and others with interest in the development 
and use of high-quality materials in science, mathematics, and 
technology education. 

Development 

The Expert Panel praised NSF for supporting the 
development of materials that support reform of 
mathematics, science, and technology education. 
Further, the members noted that materials are one 
element in efforts to bring about change in schools, 
and believed that use of the IMD-supported materials 
is likely to increase as the education community 
becomes more aware and supportive of reform. At the 
same time, the Panel raised several concerns about the 
processes recipients of IMD support used to develop 
instructional materials. The Panel members suggested 
some modifications in the development process in 
order to mitigate the difficulties encountered in secur- 
ing adoption and implementation of innovative materi- 
als, especially those designed as comprehensive 
curricula, because they believe that a number of the 
problems arose from the processes used. The follow- 
ing summarizes the Panelists’ views with regard to 
development: 

• The products that resulted from IMD support 
are of high quality when judged against stan- 
dards for mathematics and science education. 
Consequently, the Panel supports the ap- 
proaches NSF is using to select projects for 
support. 



• The Panel noted that NSF seems to hold two 
different goals for the program. On the one hand, 
NSF seems to seek the development of materials 
that can serve as ideal models of curriculum and 
pedagogy. If NSF seeks ideal models, issues of 
dissemination and use are less important than 
whether the models encourage publishers, teach- 
ers, and others to consider the implications of 
deep change in mathematics, science, and technol- 
ogy education. On the other hand, if the goal is to 
develop innovative materials and curricula that 
would be widely adopted and implemented in 
schools, then judgment about the extent of use is 
appropriate, and materials should be amenable to 
adoption within the current capabilities of schools. 
Although the Panel deemed both goals as worthy 
of support, members believed that NSF might 
consider funding each approach in a separate 
program. If NSF did so, developers would be 
guided by clearer expectations as they designed 
projects. 

• Whatever the goals of the program, the most 
successful projects were informed by the 
realities of kindergarten to grade 12 class- 
rooms. In this regard, it seems important for 
developers to include teachers at every stage of 
development, pilot, and field testing to ensure 
that their voices are heard. Further, although 
NSF may wish to demonstrate the potential 
value of technology to student learning and to 
preparing students for a life in which technol- 
ogy has a central place, but should be aware 
that many schools are not yet capable of using 
a number of technology applications. Panel 
members agreed that it was important for NSF 
to demonstrate the possibilities of cutting edge 
technology, but also noted the limited immedi- 
ate impact of such support. 

• In addition to ensuring that materials meet 
national science and mathematics standards, 
developers should include information that 
helps potential adopters relate the materials to 
state and local standards. NSF can facilitate 
this by providing developers with information 
about state mathematics and science standards 
and state-mandated assessment programs. 

• Assessment of student learning is essential for 
IMD success. The Panel believed that devel- 
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opers should provide data comparing how well 
students learn using the IMD materials, including 
how well they retain knowledge and skills over 
time, with student learning from conventional 
curricula, using appropriate standards and bench- 
marks. In addition, the materials should contain 
assessment approaches that teachers can use to 
determine how well their students are learning. 

• In addition to providing information about student 
learning, developers should engage students in 
qualitative assessments of the materials. 

• Developers should include information in their 
dissemination materials that reflects an analy- 
sis of management issues, such as the time 
required for preparation, materials needed for 
implementation, and administrative and finan- 
cial support required. Even more important, 
developers should assess the need for profes- 
sional development, both in content and peda- 
gogy, to support implementation. Such sup- 
port can come from a variety of sources, and 
NSF should increase program integration to 
ensure that professional development is sus- 
tained and comprehensive (see Implementa- 
tion, below). 

Dissemination 

The Expert Panel expressed concern about ap- 
proaches to promoting, disseminating, and marketing the 
materials funded through the IMD program. In part, the 
concern related to the perceived lack of clarity about 
program goals, because approaches to dissemination 
would differ if the goal were to create ideal models of 
curricula from what they would be if widespread adoption 
were the aim. The Panel recognizes that the insight and 
advice offered may reflect a similar lack of clarity about 
goals, but believes that its comments will be useful to 
developers and disseminators. Further, the Panel is aware 
that the publishing industry is in a phase of great consolida- 
tion, and the number of publishing outlets and actively 
marketed programs has decreased. Consequently, the 
suggestions offered below should be placed in the current 
context: 

• NSF should consider a variety of approaches, 
including amending the program guidelines, to 
encourage greater levels of commitment from 
publishers to publish and market materials emanat- 
ing from IMD projects prior to making grant 



awards. To the extent possible, the level of 
publisher commitment should be a factor in 
selecting projects, with letters expressing support 
in vague terms given limited weight as compared 
to those that outline specific steps that the pub- 
lisher is willing to take to market the materials. The 
Panel recognizes that publishers cannot make final 
decisions about materials without reviewing actual 
samples. However, publishers can play a support- 
ive role during development by working with 
authors on ways of organizing the materials to 
make them marketable. 

In order to facilitate publishing and marketing, 

NSF should provide developers and their 
publishers with information about specific 
difficulties, including legal and financial 
problems, that previous IMD projects have 
encountered. Such information could help new 
projects avoid actions that have led to past 
difficulties, thereby leading to better product 
design and marketing plans. 

The Panel encourages NSF to develop policies 
about using the World Wide Web in develop- 
ment, publication, dissemination, and support 
of IMD-funded materials. For example, 
substantive policies regarding continued 
maintenance of web-based materials for devel- 
opers and disseminators who make use of the 
Internet would enhance marketing efforts. 
Although many adopters of IMD-funded 
products were associated with other NSF 
programs, such as the State Systemic Initia- 
tives (SSI), Urban Systemic Initiatives (USI), 
Rural Systemic Initiative (RSI), and the Local 
Systemic Change (LSC) projects, a number of 
teachers, schools, and districts adopted and 
implemented products without such associa- 
tions. NSF should sponsor a study of such 
“non-associated” sites to find out what 
prompted their choice and how they experi- 
enced implementation. Such information 
could be useful to both developers and publish- 
ers. 

On a technical level, the Panel expressed 
concern that current consolidation in the 
publishing industry might place NSF-supported 
products at risk. Such risk could result from the 
merger of publishers of IMD materials with 
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companies less supportive of reform. Conse- 
quently, the Panel recommended that NSF should 
consider requiring a reversion of rights clause in 
IMD publishing contracts to cover cases of 
bankruptcy, if this is feasible. Further, the con- 
tracts should include minimal marketing require- 
ments, such as continuous inclusion of the materi- 
als in the publisher’s catalog and inventory mainte- 
nance, to the extent feasible. 

Adoption 

The Expert Panel noted the paucity of data about 
adoption and expressed three major concerns about the 
findings of the study related to adoption. First, Panel 
members noted that materials are often adopted oppor- 
tunistically, either by teachers or at the school or 
district levels. Second, they remarked that a large 
number of teachers who play a major role in curricu- 
lum selection are frequently unprepared for making 
such choices. Third, they were concerned that the 
study was unable to include data related to how widely 
the materials are actually adopted, and the lack of such 
information limited knowledge about the effectiveness 
of the materials beyond the sites identified by develop- 
ers. The Panel, therefore, offers the following sugges- 
tions: 

• NSF should support efforts that provide infor- 
mation about products aligned with reform of 
mathematics, science, and technology educa- 
tion. Teachers and others in schools are not 
aware of the full range of materials available 
and, therefore, cannot make appropriate 
choices. NSF should encourage developers 
and publishers to work with state and national 
professional organizations, state education 
agencies, and schools, colleges, and depart- 
ments of education as they disseminate their 
materials. In addition, NSF should support 
web-based dissemination activities. Develop- 
ers and disseminators who understand the 
adoption process as it actually occurs will be 
better able to offer useful information, and 
including teachers on development teams may 
enhance this understanding. 



• In order to assist teachers, schools, and dis- 
tricts to develop thoughtful, professional 
approaches to adopting materials, NSF should 
support programs that assist them in learning 
appropriate models for adoption. Figure 1 
provides an overview of an ideal model for 
adoption. In this model, district committees 
consider the needs of their communities, the 
demands of the curriculum, and the goals they 
wish to attain. Further, once materials are 
adopted, the committee develops a strategic 
plan for implementation, including sufficient 
and sustained professional development and 
community support. The model could be 
adapted at the school level in site-based man- 
agement schools. NSF should consider a 
variety of structures through which to support 
better curriculum decision making. 



Figure 1 

Suggested School District Adoption Model 
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• Whatever the approach to curriculum selection, 
teachers are likely to play a significant role in 
the process. Consequently, it is important that 
preservice education courses address the issues 
and processes involved in selected materials. 
Further, the reform-oriented materials tend to 
have strong, coherent, and challenging math- 
ematics and science as their base. To under- 
stand such material and teach it well to stu- 
dents requires teachers who are prepared with 
solid mathematics and science backgrounds. 
NSF should continue and strengthen its atten- 
tion to preservice education, including concern 
for curriculum materials. 

• NSF needs better information about the extent 
to which the IMD materials are adopted, the 
types of districts that adopt them, and the 
reasons that some choose these materials and 
others do not. Although developers, dissemi- 
nators, and some adopters claim that the 
market for reform-oriented materials is small, 
if NSF’ s goal is to broaden implementation of 
the materials, better information is needed to 
support the claim. The Panel recognized the 
difficulty of collecting such information, and 
recommended that NSF consult with publish- 
ers, researchers, and school- and district-based 
professionals to design a study that will pro- 
vide the necessary information. 

Implementation 

The findings of the Evaluation of the Instructional 
Materials Development Program continually emphasize the 
essential role of professional development to ensure 
appropriate selection and use of the materials. The Expert 
Panel underscored the importance of ongoing, sustained, 
supportive professional development, beginning with 
preservice and continuing throughout teachers’ careers. 
The members noted that the materials developed with 
IMD funding could foster fundamental change in the 
teaching-learning environment. As such, the products 
make great demands on teachers, who are faced with the 
difficulty of changing their practice. As a result, the Panel 
focused on ways NSF could facilitate and support profes- 
sional development for those who are implementing the 
materials. 

• Strong collaboration among programs through- 
out NSF would build the support structures 

O 



that will enhance implementation of IMD products. 
The new IMD Implementation and Dissemination 
Sites have a potentially strong role to play in 
building such support. Further, recognizing the 
role of developers in disseminating and facilitating 
implementation, NSF should create incentives and 
processes that put their knowledge and skill to 
good use. NSF-sponsored programs in 
preservice education, as well as teacher enhance- 
ment projects and the various systemic initiatives, 
can be part of the support structure. 

• Because NSF has a number of programs that 
could provide professional development to 
facilitate implementation, the Panel urges 
agency staff to continue to explore ways to 
integrate program efforts effectively. 

• NSF should explore a variety of additional 
ways to provide the necessary professional 
development. The Panel notes that particularly 
powerful models build on the knowledge and 
credibility of teachers who are already imple- 
menting the materials and involve them in 
networks of users. In addition, support 
through 24-hour help lines and the web may 
prove useful. 

• Implementation is enhanced by community 
support for the use of the materials. Conse- 
quently, the Panel recommended that NSF 
encourage projects to devise materials and 
activities that build parent and community 
support, through demonstrations, parent- and 
community-oriented materials, and the use of 
technology. 

• Implementation is also enhanced when teach- 
ers have strong subject-matter backgrounds. 
The Panel noted that preservice and inservice 
education that strengthens content knowledge 
will support increased use of standards-based 
instructional materials. 

• Developers and publishers should be clear that 
implementation is a difficult and uneven 
process. Such clarity will create appropriate 
expectations, and help teachers, schools, and 
districts persist during periods in which they 
are experiencing problems. 
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Impact 



Conclusion 



The Expert Panel emphasized that the develop- 
ment of materials is the first step in a long process. 
Consequently, the members believed that NSF should 
continue to fund studies that document the implemen- 
tation and impact of the materials. The Panel also 
returned to the question of IMD program goals in their 
discussions of impact. From the perspective of Panel 
members, the current study’s examination of develop- 
ment, dissemination, adoption, implementation, and 
impact is an appropriate approach for materials de- 
signed for widespread use. In contrast, panelists 
believed that materials created to exemplify cutting 
edge approaches to mathematics, science, and technol- 
ogy education should be judged in terms of how well 
they meet standards and not on their use. However, 
with either goal, Panel members agreed that better data 
about student learning essential. The Panel suggests the 
following: 

• The Panel noted that where the materials were 
well implemented and data related to instruc- 
tional approaches and student learning were 
collected, the materials had a positive impact. 

• Measures of the impact of the materials should 
focus on student learning. The key question to 
ask is the extent to which appropriate use of 

the materials enhances student learning of 
important content in mathematics, science, and 
technology. 

• The Panel identified a number of questions that 
NSF should ask as it assesses impact: 

1 . Do the materials change how mathematics, 
science, and/or technology is taught and 
learned? 

2 . Are the learning outcomes strong enough to 
show an impact and convince publishers to 
adopt similar approaches and/or districts to 
adopt the materials? 

3 . Does the technology used in a project improve 
learning and expand appropriate use of technology 
in schools? 

4. Is there sufficient and appropriate staff develop- 
ment to make the project worthwhile over 
time and extend its benefits beyond initial 
users? 

5 . Does the project fit in sufficient numbers of real 
school situations to allow it to survive 

long enough to have the intended impact? 



The Expert Panel believed that the IMD program 
has produced high-quality materials, and the ones 
included in this study reflect national standards in math- 
ematics, science, and technology education. The insights 
and suggestions they offered were intended to improve the 
program. Indeed, Panel members urged NSF staff to 
view this evaluation as a mid-course, formative evaluation. 
In that regard, the panelists recommended that NSF 
sponsor a similar study in five years, when many of the 
materials will have been available and used for a sufficient 
period to assess their impact on teaching and learning. 

Panel members emphasized that a successful ap- 
proach to developing materials that will support and 
facilitate reform should be marked by some key 
elements: 

1 . Content and pedagogy based on the standards 
of the field and what is known about effective 
approaches to instruction; 

2. Assessment of student outcomes; 

3. Appropriate use of technology, including 
delivery using the World Wide Web; 

4. Structures and processes that ensure that 
developers and publishers understand the 
environment of public school teaching; and 

5. Long-term, sustained staff development. 

The Panel members believed that some of these 
elements are currently included throughout the IMD 
program and others are included in specific projects 
and should be continued and strengthened. Further, 
the panelists suggested NSF coordinate existing 
programs so that preservice and inservice teacher 
development and the use of IMD materials work 
together to enhance reform of mathematics, science, 
and technology education. 

Summary of Findings 



Content experts affirmed that the IMD materi- 
als reviewed for this study embody the national 
standards for science and mathematics and reflect 
current thinking about best instructional practice. 

All projects used a development process that 
was well grounded in research and involved the 
participation of a variety of individuals represent- 
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ing different professional positions and areas of 
expertise. The involvement of teachers was 
particularly important. 

Although the development process involved field- 
testing, evaluation, and revision, few projects included 
assessment of student learning in the evaluation. 

Rather, the focus was on ease of use and student 
engagement. * 

For virtually all products, the field-test sites 
became the initial target market. 

Large publishers consider the market for * 

reform-oriented materials to be narrow and 
controversial, and they tended to shy away from 
carrying IMD products. 

Effective marketing depends on developers and 
publishers sharing a vision of the products and the * 
role of materials in advancing the reform agenda. 

Marketing was most effective when it involved 
professional development in the form of in-person 
seminars and hands-on workshops. 

Both developers and publishers agreed that 
NSF should engage in a major communication 
campaign with teachers, decision makers, and the 
public to encourage educational reform. This 
would, they believed, expand the market for IMD * 
materials. 

Fewer adoptions, even of comprehensive 
materials, were made by district or school teams 
and more by individuals than was anticipated. 

The most successful approaches to adoption 
involve processes and criteria that foster teacher 
investment in and parental support for the materi- 
als. 

The availability of professional development is 
an important influence on adoption decisions. 

The materials challenge teachers to reform 
instructional practices, which many teachers find 
difficult. The reform emphasis may stimulate 
both teacher and community resistance, but also 
may improve the teaching-learning environment. 
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The presence of visible advocates in the district or 
school enhances implementation. 

Successful implementation, particularly of compre- 
hensive curricula, is supported by sustained profes- 
sional development, including introductory and 
refresher workshops and on-site support. 

Elementary school teachers, especially related 
to science, believe the materials increased their 
content knowledge, in sites in which the materials 
were well implemented. 

The materials stimulated the use of student-led 
investigations and discovery activities, hands-on 
exercises, and exploration of mathematics and 
science concepts in sites in which the materials 
were well implemented. 

Most teachers reported high levels of student 
engagement in mathematics and science in classes 
in sites in which the materials were well imple- 
mented. A few materials were criticized for 
having reading levels that were too difficult for 
the students. 

Although data related to student achievement 
are limited, existing information shows positive 
outcomes. 

Overall, the IMD program produces high- quality 
products. However gaps between program and 
project intention and actuality appeared at every 
transition point — as the materials moved from devel- 
opment to publication; from publication to marketing; 
from marketing to adoption; and from adoption to 
implementation. When developers and publishers 
shared a vision about the material, when the materials 
were marketed in ways that facilitate understanding of 
the intent of the products, and when adopted in ways 
that build investment, and when implemented with the 
support of sustained professional development, the 
materials have a positive impact. 
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Introduction 

The National Science Foundation (NSF) has provided funding for systematically devel- 
oped, research-based curriculum materials beginning in the 1960s. Over the years, there have 
been changes in the levels of funding for such instructional materials, reflecting changes in public 
support and educational concern for such endeavors, with the low point following congressional 
criticism of Man, A Course of Study (MACOS), an NSF-funded anthropology-based curricu- 
lum. More recently, however, concerns about student achievement in mathematics and science 
have focused attention on the need for strong curriculum materials (Flanders, 1987; McKnight, 
Crosswhite, Dossey, Kifer, Swafford, Travers, & Cooney, 1987; O’Day & Smith, 1993) to 
support “systemic reform.” 

NSF has responded to these needs by increasing attention to research-based instructional 
materials. Materials developed through NSF funding have been reviewed by content experts on 
at least two occasions. Both groups found the materials to be of high quality and meet the 
demands of the National Council of Teachers of Mathematics (NCTM) and National Science 
Education Standards (NSES). 

Questions remain about the adoption and use of such materials. Some research indicates 
that teachers and others are more likely to use materials developed locally than those developed 
by experts (Fullan, Anderson, & Newton, 1986). Other studies show that teachers and school 
and district decision makers are concerned with the quality of materials, and that, at least for 
some, quality is judged by some “external” development or validation process (Louis & 
Rosenblum, 1981; Crandall et al., 1982). Most recently, Slavin has argued that the seeming 
failure of educational reform is rooted in the lack of clear and well-developed models for class- 
room instruction (Slavin, 1997). Carefully developed, research-based curriculum materials, 
such as those sponsored by NSF, could be seen as meeting the need for such models (Schmidt 
et al., 1996). 

Recognizing a lack of information about the adoption and use of NSF-sponsored instruc- 
tional materials, two divisions within the agency funded a study of the adoption and use of 
materials developed through the Instructional Materials Development (IMD) program. WestEd 
and Abt Associates Inc. are carrying out a research study of the quality of a selection of the 
materials, the development process used, reasons for their adoption, and implementation issues 
related to teacher use. 

The IMD program supports the development of materials and strategies that promote 
improved science, mathematics, and technology instruction at all levels. The study is designed to 
answer the following questions: 

1 . To what extent do instructional materials embody the national standards, 
including an emphasis on thinking skills and making connections across 
curriculum topics? 

2. To what extent do they reflect what is currently known about good instruc- 
tional practice? 

3. How well have they been marketed? 
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4. To what extent do adopters and teachers use the materials? 

5. What supports do teachers and other school-based professionals need to 
make the best use of the materials? 

6. What is the impact of the materials on classroom practice? 

As the report indicates, the study was best able to provide solid answers to the first two questions 
posed by NSF and was more tentative about the others. The findings about the last four questions lay 
the groundwork for future NSF-sponsored studies. 

The evaluation comprised five phases: 

• An expert panel review of the quality of the materials; 

• Telephone interviews with developers and marketers; 

• Telephone interviews with key school or school district personnel; 

• Focus groups with teachers; and 

• Observations of classrooms. 

Working with IMD staff, WestEd and Abt Associates Inc. selected 30 products, including 
elementary, middle, and high school science and mathematics; full courses and supplemental 
curricula that were intended to be used only as part of a course of study; and technology educa- 
tion and use of technology in education. NSF recommended products that not only illustrated the 
types of products funded, but also represented both fairly new and more “mature” materials. 

This selection was intended to ensure that the evaluation would identify differences in ap- 
proaches to development and dissemination under evolving NSF guidelines, as well as issues 
related to adoption and implementation when products were quite new and when they were well 
established. In addition, the contractors selected 15 widely used mathematics and science prod- 
ucts that did not receive NSF funding. The “non-NSF” products were not reviewed by experts, 
nor were there interviews with developers. For those products, the study began with adopters. 

As is clear, this study was designed to answer questions about the development and use of 
products in classrooms. However, an equally valuable study could be based on the premise that 
the purpose of the IMD program is to provide models of curricula for a variety of audiences. 

From that perspective, an evaluation study would focus on how IMD products, taken as a whole, 
influence mainstream publishers, developers of state and local guidelines for adopting materials, 
and pre- and inservice educational opportunities. The alternative study of the influence of the 
products would start from the premise that the value of federal funding for curriculum materials 
lies in providing the field with “ideal” examples, rather than solely on their use in classrooms. 
Indeed, we believe that NSF should sponsor a study based on an image of the role of the materi- 
als in facilitating a vision of mathematics and science education rooted in conceptions of educa- 
tional reform. 

Whatever the merit of a study of product development as an instrument for presenting 
ideals to the field, it was not the question posed by NSF for this study. Rather, NSF was inter- 
ested in actual use of products, including an analysis of the barriers and facilitators of such use. 
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Evaluation Methods 



The IMD program evaluation was a mixed methods study, based on a design for gathering 
information at each step in a chain from product development to classroom use. It combined an expert 
panel review of curriculum materials, telephone interviews, focus groups, and classroom observations. 

In addition, the expert panel served as interpreters of the meaning of the study. This section will begin 
with a description of the overall design of the study and then move to a discussion of each method used, 
the data collected through its use, the evaluation questions answered, and the rationale for the choice of 
method. 

The IMD program evaluation was designed to answer questions related to the development, 
dissemination, and use of NSF-supported materials. The questions reflect concerns at each stage 
of development and use, starting with issues related to the quality of the materials and ending 
with their application in classrooms. The design was intended to inform NSF and others about 
the relationship of the development process to quality, and how both quality and approaches to 
marketing affect adoption and use. Further, the design provided contrasting information about 
marketing, adoption, and use through our identification of widely used non-NSF products and 
how they were disseminated, adopted, and used. Figure 2 shows the evaluation framework. 

Figure 2 

Evaluation framework 



IMD 




Non-IMD 




Each approach to data collection was designed to answer questions about a specific stage in 
the chain from development to use. This approach is in contrast to some multimethod studies in 
which multiple sources of data are used to answer one question. Table 1 presents the evaluation 
question, data sources, and methods. 
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Table 1 

Evaluation question, data source, and method 



Evaluation Question 


Data Source 


Method 


To what extent do instructional materials 
embody content standards and current 
knowledge of effective instructional 
practice? 


Expert panel review 
(IMD) 

Developers (IMD) 


Structured written 
review 

Telephone interview 


How well have materials been marketed? 


Marketers (IMD) 


Telephone interview 


To what extent are users of IMD and non- 
IMD materials satisfied with them? 


Decision makers 
Classroom teachers 


Telephone interview 
Focus group 


What supports do teachers and other 
school-based professionals need to make 
the best use of the materials? 


Classroom teachers 


Focus group 


What impact do the materials have on 
classroom practice? 


Classroom teachers 


Observations 



Development 

Information about the extent to which the materials embody national content standards, 
including an emphasis on thinking skills and making connections across curriculum topics, comes 
from two sources. First, the Expert Panel reviewed the products, using a structured format that 
directs attention at key quality concerns. Content and pedagogical experts brought to bear their 
knowledge and judge the output of the program. Their ratings (and subsequent discussions) 
provide indicators of project and program effectiveness. 

The members of the Expert Panel (see Appendix 1) were identified by NSF and included 
scientists and mathematicians; science, mathematics, and technology educators; a former pub- 
lisher; school and district administrators; and classroom teachers. WestEd and Abt Associates 
Inc. adapted an instrument for evaluating the quality of the materials that was previously devel- 
oped by Inverness Research under contract to NSF for a study of year-long courses in middle 
school science (see Appendix 2). The adaptation was designed so the instrument would be 
applicable to the broader range of materials included in this study. 

The Expert Panel met to explore its role in the study and develop shared understanding of 
the instrument used to evaluate the materials. The meeting included an opportunity to rate some 
products not included in the study and come to agreement on the meaning of each element in the 
rating form. Then, two members of the panel with relevant background rated each product. The 
group met again to resolve differences in the ratings and share their views on the quality of the 
materials. 

Second, interviews, lasting from 45 minutes to over an hour, with key members of the 
product development teams of the projects identified by NSF included questions related to the 
intellectual underpinnings of the product, including bodies of scientific, cognitive science, and 
pedagogical knowledge used to inform their work. The developer interviews also included 
questions about the development process, including formative and summative evaluation efforts. 
The interviews not only reconstructed the development process, but they also provided develop- 
ers with an opportunity to reflect on their experience and how they might approach the task in the 
future. During the interviews, we also asked developers to share supporting documentation from 
their project, including evaluations, field-test results, annual reports, and dissemination materials. 
As a result, the study made extensive use of already existing information. 




Final Report on the Evaluation of the National Science Foundation’s Instructional Materials Development Program 

15 



Dissemination 



High-quality materials must be appropriately disseminated to ensure their impact on teachers and 
students. Consequently, the evaluation included a focus on marketing and dissemination. 

We gathered information about dissemination through telephone interviews with publishers 
and distribution houses that package and disseminate kits and other materials for EMD develop- 
ers. The non-IMD products were identified through a review of adoption lists in states that have 
such a process, because we believed that materials included on the list would have widespread 
use. Our original intention was to conduct similar interviews with the publishers of non-IMD 
developed materials, but those publishers refused to participate in the study in any way. Conse- 
quently, the final report does not include their perspective on marketing. As a result, we cannot 
compare and contrast the marketing approaches of IMD and non-IMD publishers. 

Interviews with publishers were the method of choice for two reasons. First, they allowed 
us to probe for the reasons that particular marketing decisions were made. Second, given the 
reluctance of commercial vendors to release information related to sales, telephone interviews 
seemed less threatening than written questionnaires or surveys, or even in-person interviews. 
Telephone interviews are also cost effective. 

Adoption 

Cost effectiveness was the major reason for using telephone interviews of adopters, the 
customers for the products. Although the original design called for a random selection of adopt- 
ers from lists provided by publishers, the publishers would not share that information. (Nor 
would they share information about the numbers of adoptions.) As a compromise, we asked for 
names of “some” adopters, allowing the publisher to select those to include, but pressing for 
variation in type of setting and students served. From these lists, we selected schools and dis- 
tricts throughout the nation, representing rural, urban, and suburban settings. This approach is 
less desirable than randomly sampling adopters because it is likely that the names we receive represent 
“good adoptions” — districts that participate in publisher- and developer-sponsored staff development, 
purchase related materials, and provide feedback to the marketer. 

The adopters of non-IMD products were identified by asking the IMD adopters to identify 
nearby schools or districts that used more mainstream products. Although we attempted to select 
adopters in areas with demographics similar to those in the IMD group, we were not always able 
to do so. 

The interviews yielded information about the process and criteria used for adoption. Further, 
adopters reported on their use of the materials. Adopters of both IMD and non-IMD products were 
interviewed so we could see if there are particular processes and criteria associated with the selection of 
IMD products. Further, differences in use and impact may emerge. 

Classroom Use 

We elected to conduct focus groups of classroom teachers to gather information about their 
use of the materials. Focus groups are an efficient method to gather information on specific 
topics defined by the evaluator. The evaluator can direct the group in a manner that encourages 
interaction and reveals much about how consumers use the materials. The focus groups explored 
both strengths and weaknesses in the materials through the eyes of the users, as well as encour- 
aged discussion of how teachers were (or should be) supported through workshops and other 
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professional development opportunities. Further, teachers were asked to reflect on how the use of the 
materials affected their classroom practice and student learning. 

We were not always able to form focus groups, particularly for supplementary products. In those 
cases, we conducted telephone interviews with users. 

Finally, classroom observations, using an instrument that reflects content standards in 
mathematics and science and knowledge of effective instructional practice, enabled us to contrast 
classroom practice and student response to IMD and non-IMD materials (see Appendix 3). 
WestEd and Abt Associates Inc. staff members conducted the observations. Only classrooms 
using full-course products were observed because the logistics of scheduling observations of supple- 
mentary materials in a variety of settings across the nation was an overwhelming prospect. Such materi- 
als are frequently used on an ad hoc basis, and teachers cannot predict when they will be drawing upon 
them. 



Summary 

The study employed a variety of methods to examine issues at each step from development 
to use. The design assumes that the purpose for developing materials is for them to be used in 
classrooms, although it does not assume that IMD materials should dominate the market. In fact, 
our approach will enhance NSF’s understanding of the circumstances under which IMD-sup- 
ported materials lead to high customer use, as well as affect classroom practice. Such under- 
standing may be just as useful for the design of dissemination and professional development 
programs as for materials development programs. 

From an evaluation methodology perspective, the study extends the use of multiple method 
approaches to evaluation. Most earlier multimethod evaluations are either integrative or sequen- 
tial (Caracelli & Greene, 1997). Integrative studies use a variety of methods to illuminate a 
complex phenomenon by employing such devices as consolidated coding, concept mapping, or 
nesting qualitative methods in an experimental study. Sequential multimethod evaluations 
typically use qualitative methods to gather information that helps in the design of quantitative 
instruments, which are then used to confirm or extend findings. This study, however, uses 
various methods in a sequential fashion in order to build an integrated picture of the develop- 
ment, dissemination, and use of instructional materials. 

Limitations of the Evaluation 

As with most evaluations, the implementation of the design faced some realities that limit 
the strength of the findings. The major limitation of this study has been alluded to above — and 
that is publishers’ (both of IMD and non-IMD products) refusal to share information with us. As 
a result, the evaluation cannot report on market penetration. It also cannot include an analysis of 
similarities and differences between approaches to marketing by the publishers of the two differ- 
ent types of products. 

Publishers of IMD materials were willing to speak with us about the issues they faced in 
marketing the products, and they also provided names of some adopters. Their lists enabled us to 
select adoption sites that varied in demographic qualities, including urban, rural, and suburban 
districts. However, we are cautious in generalizing the findings about adoption and use because 
the sites in which we conducted interviews, focus groups, and classroom observations were 
sampled from what is likely a biased list. 
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Organization of the Report 



The final report of the IMD study is organized around the questions originally posed. It 
begins with an analysis of the IMD products that are included in the study, starting with the 
assessment of their quality by the Expert Panel, as well as descriptive information about them. 
The report then moves to a focus on the development process, followed by a discussion of 
marketing and dissemination. We then turn to adoption, including an analysis of approaches to 
adoption that seem most likely to affect implementation. Implementation itself is the focus of 
the section that follows, and, along with providing information across products, the section looks 
at issues that facilitated or impeded successful implementation. 

The concluding section summarizes the findings, and provides recommendations to NSF that are 
intended to enhance the impact of the IMD program. 



Science Foundation 



7 



CHAPTER II 



The Products 

The study focused on 30 products developed with IMD funding, selected by NSF staff to 
reflect a variety of characteristics. Seven products were designed for elementary schools stu- 
dents, including three in mathematics and four in science; twelve were designed for middle 
schools students, with five mathematics, six science (including two that were science and tech- 
nology), and one technology; and eleven for high school, including five mathematics and six 
science. Further, the materials varied from full course, sometimes covering multiple grades, to 
supplementary materials. NSF also included materials that received funding fairly early, and 
some that were more recently funded in order that we might see differences in development, 
marketing, adoption, and use between recent and more mature programs. Differences also 
existed in the extent to which the products used instructional technology and the types of tech- 
nology they used. (Throughout the report, the products are referred to as “Project x,” with a 
number replacing the product’s name. The numbers were assigned randomly to maintain the 
confidentiality we promised developers, marketers, adopters, and teachers.) 

Tables 2-4 list the products by grade level and content area, with (S) indicating that the 
materials are supplementary. 



Table 2 

Elementary school products included 



Mathematics 


Science/Technoloev 


TERC 


INSIGHTS 


Everyday Mathematics 


Science and Technology for Children 




(S) 


MathTalk (S) 


Full Option Science 




Project ARIES (S) 



Table 3 

Middle school products included 



Mathematics 


Science/Technolosv 


Middle School Mathematics Through Applications (MMAP) 


Science and Technology (BSCS) 


Math in Context 


Science and Technology 
(IMaST) 


Connected Math 


SEPUP 


Jasper Woodbury (S) 


Science 2000 (S) 


Sky Math 


Middle School Life Science 



A World in Motion (S) 
Random Universe (S) 



Table 4 

High school products included 




Mathematics 


Science/Technoloev 


Interactive Math Project 


Adapting Beyond the Mechanical Universe 
(S) 


Core Plus 


Biological Sciences: A Human Approach 


Project Mathematics 


Chemistry in the Community 


Visual Geometry 


ARGUS (Geography) 


Data-Driven Curriculum Strand (Statistics) (S) 


Fast Plants (S) 

Human Genome Project (Videodiscovery) 
(S) 
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We selected the non-IMD products from the lists generated by state adoptions, because we 
believed that such materials would be widely used, which would allow us to pursue adoption and 
implementation issues in schools and districts with similar characteristics to the ones using IMD 
products. The non-IMD products also included one curriculum developed by a school district, to 
enable us to gain some insight into differences between the effects of external and local develop- 
ment. The products were: 

• Local district-developed elementary science, K-8 

• Scholastic Science, K-3 

• Prentice-Hall Middle School Science, 6-8 

• Honors High School Chemistry (used a combination of texts, including ones 
published by Prentice-Hall and Glencoe) 

• Saxon Mathematics, K 

• Creative Publications Mathematics, 2-3 

• Addison-Wesley Middle School Math, 6-8 

• Houghton-Mifflin Mathematics, 9 

• Houghton-Mifflin High School Geometry, 10 

Content Quality 

Experts and users judged the IMD materials to be of high content quality, although materials held 
up less well on those criteria related to implementation. This section provides information about the 
assessment of quality from each perspective, with some hypotheses about the reasons for differences. 

Expert Panel 

The panel of national experts affirmed that, overall, the IMD materials included in this 
study embody the national standards and successfully reflect current thinking regarding best 
instructional practice. The materials are designed to promote critical thinking and problem 
solving skills and address the needs of all students regardless of background, gender, or ability 
level. They make connections to real-world topics that span and integrate various subject 
matter areas. 

The content experts used measures of quality based on the National Council of Teachers 
of Mathematics (NCTM) standards and the National Science Education Standards (NSES) to 
review the materials. Each set of materials was reviewed by two experts and discussed within a 
larger group. The panel reflected knowledge of science, mathematics, and technology, as well 
as expertise in pedagogy and learning. The reviews were largely positive; the experts scored 
most items above 3.5 on a 5-point scale — 5 being the highest score (e.g., “the image of science 
is current and accurate,” “high overall quality of science/mathematics”), and 1 the lowest rating 
(e.g., “the image of science is out of date, inaccurate, or non-existent,” “low overall quality of 
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science/mathematics"). Tables 5 and 6 presenrkey findings (high-and low-rated items) for science 
and mathematics materials developed under IMD support. 

Table 5 

Key findings in science 



Questions 


Score (using a 
5-point scale) 


Do the materials provide sufficient activities for students to develop a 
good understanding of key science concepts? 


4.5 


Do the materials accurately represent views of science as inquiry? 


4.4 


To what extent do the materials provide students the opportunity to 
make conjectures, gather evidence, and develop arguments to support, 
reject, and revise their preconceptions and explanations? 


4.4 


Do the materials include information and guidance to assist the teacher 
in implementing the lessons? 


4.3 


Does the content align with all 8 areas of content standards as described 
in the National Science Education Standards? 


3.7 


Do the materials provide information about the kind of resources and 
support system required to facilitate district implementation? 


3.6 


Do the materials provide information about the kinds of professional 
development experiences needed by teachers to implement them? 


2.8 


Do the materials provide guidance in how to link them with district and 
state assessment frameworks and programs? 


2.2 



Final Report on the Evaluation of the National Science Foundation's Instructional Materials Development Program 



Table 6 

Key findings in mathematics 



Questions 


Score (using a 
5-point scale) 


Are mathematics concepts accurate and correct? 


4.7 


. - 


Do the materials provide sufficient activities for students to develop a good 
understanding of key mathematics concepts? 


4.4 




Do the materials provide sufficient opportunities for students to apply their 
understanding of key mathematics concepts? 


4.2 


.^|p 


Do the materials accurately represent views of mathematical problem solving? 


4.1 




Do the materials emphasize mathematical reasoning? 


3.9 




Do the materials reflect current knowledge about effective teaching and learning 
practices? 


3.9 




Do the materials include information and guidance to assist the teacher in 
implementing the lessons? 


3.7 


, vrr* 

V4|||| 


Does the content align with all 1 3 areas of the curriculum standards in the 
NCTM Standards? 


3.4 




Do the materials provide information about the kind of resources and support 
svstem required to facilitate district implementation? 


2.6 





As noted throughout the report, the implementation stage was the most problematic in the 
process. The content experts noted that the materials were weakest in their written guidance for 
how best to provide professional development, gather community support, and align the products 
with local and state standards, all of which are essential for strong implementation. Further, we 
found that, for the most part, neither developers, publishers, nor districts mitigated the problem 
through in-person assistance on these issues. Those who did so had the most successful imple- 
mentations. 



Users 



Although users agreed with content experts that the materials met the content standards 
espoused by the curriculum organizations, they brought additional criteria for quality to bear 
upon them. From the perspective of teachers, “usability” was equally important as the elements 
of the national standards. And, although the product developers paid great attention to usability. 
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the classroom teachers in the focus groups raised questions about whether the products were, in 
fact, usable in classrooms. 

The usability criterion had three major dimensions. First, teachers were concerned about 
the timing and pace of instruction. For example, at least two products at the high school level 
were judged as requiring greater reading ability than the students had. As a result, teachers made 
a number of accommodations to students’ inability to read the texts, including in-class reading 
and pairing students. However, these adjustments slowed the pace of instruction to a point that 
raised teacher concerns. Another product contains units that do not map onto schools’ normal 
divisions, such as grading periods or semesters. Further, its content did not neatly reflect require- 
ments at the middle school in a number of states. 

A second dimension of usability relates to the use of technology. Products that rely heavily 
on technology have limited usefulness to teachers in many schools, although this may change as 
more schools gain access to computers. However, even with increased access, there is little 
likelihood that many teachers will have the number of stations one product required. Some 
teachers who used that product without sufficient computers simply copied pages from the 
curriculum and had students use them as worksheets, rather than interactively with a computer. 

The third dimension involved the degree to which the materials required changes in teach- 
ers’ classroom practices. For the most part, the IMD-funded products demand fundamental 
changes in how teachers view content, moving from a perspective that emphasized “coverage” to 
one that focuses on student understanding; how they view their role as teachers, moving from a 
perspective that puts teaching in the center to one that puts learning at the core; and how they 
view students, moving from a view of students as recipients of knowledge to one that emphasizes 
their role in creating understanding. All these changes are difficult. To the extent that NSF- 
funded materials stimulate such basic change in practice, they will support efforts to reform 
science and mathematics education. However, as will be seen, materials are not enough, and 
teachers need a wide array of support mechanisms to implement the materials in ways that 
support student achievement of new standards. 

Articulation 

The products included in this study raised questions about articulation between and among 
NSF-sponsored materials. Although it may be an artifact of the products the IMD staff selected 
for inclusion in the study, there seemed to be a heavy emphasis on middle school. Further, a 
number of teachers mentioned their concern that students came to their classrooms without the 
necessary preparation for the type of curriculum and instruction embodied by the NSF products. 
This concern was particularly strong at upper grade levels where teachers believed they had to 
“retrain” students who were seeking “correct” answers or algorithms to memorize because they 
were used to having them. Teachers at lower grade were sometimes worried that students who 
completed curricula in their classrooms would not be prepared for the demands at higher levels. 
One student said, “This curriculum prepares us for life, but I’m not sure it prepares us for high 
school.” Similar concerns were raised at the high school level about readiness for college work. 

Non-IMD Products 

Although the Expert Panel did not assess the quality of the non-IMD materials, staff used 
the same criteria to make judgments about them. Staff placed the non-IMD materials on an 
informal scale ranging from low-orientation to reform to “reform oriented.” As will be seen, 
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some of the same implementation issues arose with the more reform-oriented non-IMD materials 
as with the NSF-supported materials. 
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CHAPTER in 



Development 

This section presents key findings about the process by which the 30 EMD products were devel- 
oped. The original study design included parallel data collection and analysis related to the non-NSF- 
funded materials, but the publishers of those materials refused to cooperate with the study. Nonethe- 
less, the findings about efficiency and effectiveness of development of IMD materials provide important 
information to NSF. 

The section begins with an examination of the processes used to develop the materials, 
including the nature of development teams, approaches to pilot and field testing, and challenges 
faced by developers. 



The Development Process 

Almost without exception, the development and field-testing processes were iterative and 
intertwined. Early versions would be drafted, piloted (most often with teachers from the local 
area), and revised based upon their feedback. Then, more refined versions of the materials were 
field tested in multiple national sites and revised still further before being turned over to publish- 
ers. 

Here is a fairly typical development and field-testing process as exemplified by one project: 

1. Identify the important content to be covered within the larger area (e.g., within 
the geometry segment). 

2. Brainstorm about types of stories or powerful concepts that could be featured. 

3. Meet with the writers and review the content with them. Generate a first draft. 

4. Send the treatment out multiple times (“Drafts went back and forth ad nau- 
seam. We had at least 4 garbage cans full of old drafts!”) to the advisory 
board for feedback regarding content, dialogue, cultural sensitivity, and 
gender equity. 

5. Conduct focus groups of teachers, scholars, and community groups to review 
the treatments. 

6. Pilot an early version of the module in local classrooms. 

7. Revise module if necessary, based on pilots. 

8. Field-test final version of module in national sites, revising as necessary. 

The Development Team 

The overall approach to product development relied heavily on the composition of the 
development teams. All projects involved a wide variety of individuals representing many 
different professional positions and areas of expertise in the development process. Core teams 
usually consisted of three to five members, but could include anywhere from 20 to 400+ expert 
advisors and editors. These teams worked collaboratively to develop and test the materials. An 
impressive array of content and pedagogical expertise was typically represented within each 
product’s development team. Most frequently, these areas included: 

• Science and/or mathematics 

• Science and/or mathematics education 

• Developmental and/or cognitive psychology 

• Curriculum development 
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• Classroom teaching experience 

• Technology and multimedia production 

The involvement in development of a broad range of perspectives afforded clear benefits 
with respect to the accuracy and thoroughness of the content covered by the materials, as evi- 
denced in the expert review of the products. However, only half of the projects included teach- 
ers, either using release time, summer work, or a semester- or year-long leave, as members of the 
development team. 

A few developers believed that broad-based involvement of stakeholders helped to diffuse 
potential controversy and gather support for the products. 

“Development is a very tough process, because you have to please multiple 
audiences that all have different interests — teachers, state people, political groups, 
etc. A valuable lesson for us was to have a national advisory board. You can 
defend the content because of having had their approval.” (Developer Interview, 

Project 1 ). 

“It [development] was a very broad-based process... Each revision involved input 
from an advisory board that included teachers, people from industry, laboratories, 
and professors. ..One thing we learned is how political this process is, especially if 
you are trying to do something different. It is not enough to do well and throw it 
out there. You have to work with and build a constituency. You must be vigilant 
that things can go wrong; you need to stroke, explain, and have patience.” (De- 
veloper Interview, Project 2). 

A little over half of the development teams included classroom teachers, and both develop- 
ers and publishers of the products emanating from such teams repeatedly highlighted their 
importance in ensuring that materials would fit real-world teaching contexts. They agreed that 
extensive involvement of classroom teachers in the development process was beneficial, and the 
consensus was that a lot of involvement is good; more is even better. 

“All of us had lived in a university research environment for a long time. ..I can’t 
overstate how important their [teachers’] contribution was. One teacher we hired 
half-time and another 3/5 time. They’d teach and be with us on a daily basis. It 
was like having a school walk through the door.” (Developer Interview, Project 

7). 

“It is important that teachers be involved with the development of new materials. 

If materials do not make sense to teachers and are not able to be customized by 
them, teachers will not use them.” (Developer Interview, Project 5). 

When development teams did not include teachers, it was more likely that the products 
would fail to meet teachers’ criteria related to “usability.” For example, two products included 
“humor” that implementing teachers reported as escaping students. Others required reading 
levels far higher than students were able to achieve: 

The most vexing problem to the teachers is the reading level of the text. The 
students, in general, do not seem able to read with comprehension. Teachers say 
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they are spending more of their time teaching basic reading and comprehension than 
anything else. The second problem students have had is that they are so frequently 
absent that they fall behind and cannot catch up. Falling behind has greater conse- 
quences than usual because of how the text structures group work. Teachers wonder, 

“how are the absent students supposed to fit back in when they come back?” (Sum- 
marized from Focus Group, Project 12). 

In addition, the products that were developed without teacher input at the earliest stages 
were more likely to demand more understanding of science or mathematical content than most 
teachers possessed. 

The Research Base 

Without exception, materials were well informed by research on cognitive development, 
child development, and learning processes. In some instances, materials developed under IMD 
support were an extension of previously developed curricula or the developer’s own research on 
curriculum, instruction, and the learning process. For others, the IMD program provided the 
impetus or initial opportunity to produce innovative instructional materials that embody prin- 
ciples of learning that have emerged from work in fields such as developmental psychology and 
cognitive science. 

As evidenced by the high marks given the products by the expert panel reviewers, the 
purposes, design, and intended uses of the materials developed under the program are in keeping 
with the larger standards-based reform agenda. Without exception, developers were quick to 
offer examples of and clearly demonstrate how the content and pedagogy of their products are a 
direct reflection of the principles underlying reform, such as inquiry-based learning, 
constructivism, anchored instruction, and authentic assessment. 

[The product] is standards-based. The developers infiltrated the standards-writing 
process. Its units are cited as an exemplary way of meeting more than half of the 
standards-54 out of 72. One state listed it as one of the exemplary units for its 
standards. (Summarized from Developer Interview, Project 11 ). 

“To maximize student understanding of the content, math should arise from 
realistic situations so that the students have a need and a reason for doing the 
operation. That was the primary theoretical focus, but we also built in problems 
that encouraged student interaction, which is in line with social constructivism. 

For example, in introducing students to similar triangles and the notion of slope, 
the math problems might be presented in the context of hang gliding or building a 
bridge. Activities are organized from less formal to more formal, and they begin 
with what students already know about the particular problem being posed.” 

(Developer Interview, Project 15). 

Students Served 

In keeping with the tenets of reform, products were designed with “all” students in mind 
within the particular grade level(s) targeted. 

It is intended to be used by all high school students in grades 9-11, and has also 
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been used with high-ability 8th graders. Materials were designed to increase the 
performance of students with a variety of aptitudes and talents. The curriculum allows 
students to look at material in many different ways. (Summarized from Developer 
Interview, Project 16). 

“Our first concern was that the materials wouldn’t reproduce the bias toward 
serving suburban communities. ..So for example, a unit might ask kids to explore 
cracks in the sidewalk to see what grows there, as opposed to talking about pond 
life.” (Developer Interview, Project 10). 

One project’s attempt to level the playing field among science-oriented students and other 
students received mixed reviews from consumers: 

“Instead of the traditional sequence, we focused on societal issues that involve 
[science]... The intent was to design a high school... course for students who are 
going to college, but are not scientifically oriented. The reaction of high 
school... teachers varied. We have teachers who love us and some who don’t love 
us. Some believe that [science] must be taught in a traditional way. They might 
say that we watered it down.” (Developer Interview, Project 2). 

Another potential pitfall of attempting to target all students surfaced in connection with a 
mathematics project: instruction targeting all students that does not specifically address different 
learners’ diverse learning needs can inadvertently create the problem of “teaching to the imagi- 
nary middle.” So rather than one fitting all, it can look more like one size fitting only a few. 

One product was described as being, “developed for all students,” and yet at the same time the 
developer described the materials as targeting “the middle 60 percent of students in terms of 
achievement.” 

Comprehensive and Supplementary Materials 

Opinions regarding the relative advantage of comprehensive versus supplementary materi- 
als as a stimulus to reform arose with fairly equal incidence. Those favoring the comprehensive 
approach believed quite strongly that it represented the best way to achieve meaningful reform: 

The developers believed very strongly that the material should not be available as 
modules because it would permit schools to adopt only parts of it. They felt this 
would not bring about the complete curriculum change that they wanted. (Summa- 
rized from Developer Interview, Project 16). 

“We never considered replacement modules as an option. Always, the idea was: 

If you’re going to do it, do it. It was a philosophical decision. If you’re going to 
change how teachers teach and how children learn math, you have to change 
everything.” (Developer Interview, Project 15). 

The team moved from supplementary to comprehensive materials in part for 
reasons having to do with equity. They felt that teachers in urban schools could 
not easily buy supplemental materials, so a comprehensive curriculum would be 
more widely available. (Summarized from Developer Interview, Project 18). 
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Less strongly held views favored a supplementary approach, although the reasons for that 
position differed among respondents. Some viewed supplementary materials as being more 
responsive to the needs and realities of the school market, while others viewed the development 
of supplements as the best or at least most feasible first step toward designing comprehensive 
materials. 

“We made the decision to produce modules rather than a comprehensive 
curriculum. ..Mathematics teachers are fiercely independent and like to do things 
in their own way... By producing separate modules, the teachers would be free to 
choose whatever materials they felt would be helpful and could better integrate 
these modules into existing curricula.” ( Developer Interview, Project 8). 

The market partly directed this decision. People want to be able to pick and 
choose. Teachers need flexible, supplemental materials, and the developers 
wanted to provide something that teachers could add to their existing plans. 
(Summarized from Developer Interview, Project 12). 

Role of Assessment 

There were vast differences across products in terms of the extent to which they incorpo- 
rated student assessment within their design. Somewhat surprisingly, roughly 20 percent of the 
projects included no student assessment at all, at least not initially. 

Assessments are not included and were “not a heavy emphasis” in the eyes of the 
developers. It was assumed that teachers using the modules would “create their 
own tests.” (Summarized from Developer Interview, Project 1). 

Originally, they had no vision of student assessment. After field testing, however, 
teachers expressed a strong desire for assessment items. Developers then re- 
sponded by creating the Teacher’s Resources Package. (Summarized from Devel- 
oper Interview, Project 19). 

In the words of a developer, “We are not in the assessment business.” The prod- 
ucts carry the suggestion that students develop portfolios based on the materials, 
but there was no intention to build in assessment as part of the product. (Summa- 
rized from Developer Interview, Project 20). 

When assessments were included in the materials, nearly all tended to emphasize curricu- 
lum-specific, performance-based assessment as opposed to more traditional measures, although 
there were exceptions to this. 

Most traditional assessment, according to the Co-PI, tries to find out what stu- 
dents don’t know. Their vision of student assessment, according to all three 
interviewed, is to allow students as many ways as possible to demonstrate what 
they have learned in class, which in turn gives teachers many more ways of 
understanding what their students know and understand. They try to make teach- 
ers see that assessment is something that you do every day, and is distinct from 
grading. The teacher and student materials offer multiple modes of assessment: 



Final Report on the Evaluation of the National Science Foundation’s Instructional Materials Development Program 



oral presentations, self-assessment, problems-of-the-week, and take-home assessments 
that extend the ideas embedded within a unit to a different context. ( Summarized from 
Developer Interview, Project 21). 

“We offered suggestions for teachers on how to make observations, suggestions 
for helping kids put together portfolios and cumulative folders, and guidelines for 
helping kids decide what to showcase that would show improvement over the 
course of the module. Kids don’t consider them tests, but these activities give the 
teacher an idea of where kids are.” ( Developer Interview, Project 4). 

Pilot and Field Testing 

Development included pilot and field testing. Pilot tests were generally of early versions of 
the materials and involved teachers who were geographically close to the developers, and field 
tests included a wider variety of sites and teachers. Pilot-test teachers gave feedback to develop- 
ers, who also frequently observed in classrooms. The teachers provided information about how 
easily the materials were used, student reaction to them, and problems encountered. The value of 
the feedback is demonstrated by the changes developers made following pilot testing. However, 
we note that pilot-test teachers were drawn from those with whom developers had an earlier 
relationship, either through professional associations or prior work with the school districts that 
employed them. Consequently, they were likely to be sophisticated, reform-oriented teachers 
who had participated in earlier professional development activities. 

Products were field tested in a wide enough range of settings representing students with a 
variety of backgrounds. Local and national field-test sites included students from all ethnic 
backgrounds; ability levels; geographic areas; and from an appropriate mix of urban, rural, and 
suburban settings. There were two exceptions to this. One developer felt that too few urban 
schools were included in the field tests, which reportedly, “led to a myth that the materials were 
only good for suburban and higher level students.” The other noted that the project would have 
benefited from, “better representation in the field-test groups of high-ability and accelerated 
students.” 

The realities of school schedules, governance, and agendas often resulted in delays and 
other challenges to the field-testing process. 

Field testing suffered from many problems related to school independence. 

Schools were doing so on a voluntary basis, and implemented the materials to 
meet their own needs and commitments. Implementation varied across schools. 

Five of the ten schools put their best students into the control group and below 
average students into the project group. One school put all their students into the 
project group. Sites used different standardized tests and different testing sched- 
ules. Also, many schools did not have adequate technology to implement all the 
activities. (Summarized from Developer Interview, Project 24). 

Delays experienced in the field-testing process contributed to the widely held desire on the 
part of developers for funding to be granted for longer periods of time. 

The publisher recommends that developers be funded for a 6-7 year cycle, rather 
than the more typical 3-4. This would cover two years of research and develop- 
ment to get the materials field-test ready, three years for field testing, and then a 
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year or two to get the materials market-ready. He thinks that NSF needs to be better 
aware of the fact that curricular materials can’t really be developed any faster than 
students and teachers can try them out and developers can redesign them. (Summa- 
rized from Developer Interview, Project 6). 

The vast majority of data collected during field testing consisted of teachers’ and students’ 
perceptions of the usefulness of the materials, anecdotal evidence of students’ engagement with 
or apparent benefits received as a result of the use of the materials, and measures of teacher or 
student satisfaction and attitudes regarding science and mathematics. 

Even among the more recently funded and most commercially successful projects, there 
was a conspicuous lack of emphasis on evaluating the instructional materials’ effect on student 
learning, either immediately following the experience or to assess student retention of concepts, 
information, and skills. Indeed, some of the quotations below indicate that developers believed 
that the sole focus of the required formative evaluation was on how readily teachers could use the 
materials. While perhaps disconcerting, it is not surprising that approximately 70 percent of the 
projects conducted no evaluation of student outcomes (comparative or otherwise) that would 
enable them to measure the impact of the materials on student learning of specific content. 

“It [outcome evaluation] was not required by NSF. Measuring student gains 
wasn’t required; it wasn’t necessary. We collected qualitative information only — 
observations and comments, input from developers, teachers, trial coordinators, 
and observers. ..No other evaluation was performed. NSF doesn’t pay for assess- 
ment and data analysis. ..so the real research on the materials will have to come 
independent of us....” (Developer Interview, Project 4). 

The evaluation did not include attention to student outcomes. One team member 
did an evaluation of student attitudes early, after the first commercial edition 
(1988-89). Other results were anecdotal (e.g., impressions of the number of 
students continued in another course and were successful.). There was no concerted 
follow up to track the relative success of students. (Summarized from Documents, 

Project 2 ). 

There is constant evaluation and feedback from teachers through the developers’ 

1-800 phone number. [Name] has been the external evaluator for several years. 

While she has done a number of evaluations, they all focus on teachers rather than 
the actual material. They looked at how well the product worked as teaching 
material, with a secondary focus on students. (Summarized from Documents, 

Project 23). 

An evaluator was named in the proposal and carried out a very small scale evalua- 
tion, mostly because it was obviously required in the RFP. There was very little 
money to do evaluation — just a few days for evaluation and expenses — his role was 
more like an auditor’s. He would come in and be present at key points and would 
provide feedback to the developers on what he was hearing and observing. He 
focused on the development process, not on the content. One bit of advice the 
evaluator offers to NSF: 10 days over three years is not adequate. (Summarized 
from Evaluator Interview, Project 11 ). 
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There were exceptions — some projects used part of their IMD funding or were able to secure 
additional funding to conduct or contract for evaluations of the impact of the materials on student 
learning. Several project evaluations indicated that the materials had a positive effect. 

The developers believe that there is no purpose in developing materials that are 
unproven. If schools are going to adopt a program, there must be evidence that 
the program improves student achievement. The program received a third-party 
evaluation that indicated that students in the treatment group performed signifi- 
cantly higher than students in the control group, and that treatment students 
scored higher than the control students regardless of pretest difference in their 
achievement levels. (Summarized from Developer Interview and Project Docu- 
ments, Project 17). 

The staff completed both a formative and summative evaluation. Students who used 
the materials had higher raw scores than students in the control group. The treatment 
students answered more test questions and answered more of them correctly than 
students in the control group. On standardized tests such as the California Achieve- 
ment Test, students in the treatment group made gains of one and a half to one full 
grade-level over scores from the pretest. Attitude shifts about science were also 
monitored for girls, Latinos and African American students. Positive shifts were 
found for female students from all ethnic backgrounds. The subgroup samples of 
Latino and African American students were too small to generate any significant data. 
(Summarized from Documents, Project 19). 

“Students using the materials outperformed control group students on measures of 
complex problem-solving and did better on some of the subtests. On computation 
and application of skills, the students did as well or better than the controls.” 
(Summarized from Developer Interview, Project 1). 

Field testing involved 12 schools with 6 treatment and 6 control schools. Impact 
evaluation was done during the field testing phase. Among other impacts, the 
evaluation assessed whether there were differences in student outcomes related to 
gender, SES, or cultural background. The group’s mean achievement score was 
significantly higher than that of the comparison group, and the difference was 
significant at the .04 level. Student attitudes showed no significant differences 
between treatment and comparison groups at both the pretest and post-test. (Sum- 
marized from Project Documents, Project 5). 

Evaluation of student learning did not uniformly favor IMD products. Some materials 
evidenced little or no differences between IMD and other materials. 

The developers’ philosophy is that all activities that have measurable outcomes 
need to be assessed. Both formative and summative evaluations were done during 
both the pilot and field-test phases. A control group design was used to measure 
basic skills, higher-order learning skills, and science attitude between students using 
the materials and those in regular classes. Standardized achievement tests were 
used along with problem-solving measures. The major result of the evaluation 
was that there was no significant difference on content knowledge gained 
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between students using the product and those in regular classes. (Summarized from 
Interviews and Documents, Project 24). 

Development and Marketing 

For virtually all products, the field-test sites turned out to serve as initial target markets. 
The involvement and participation of teachers as product developers led quite naturally to their 
buy-in and subsequent interest in adopting the materials. 

“...what they did was to bring together teachers to participate in training work- 
shops during the field-testing even before the products were published. This 
created a cadre of people who keep coming back. It’s been a fabulous way to get 
a stronghold in the market.. .You’ve got ready customers out there because of the 
field testing, and those contacts require care and feeding. You’ve got to manage 
the demand that the field testing creates, as opposed to just waiting to recontact 
them two or three years later.” (Summary from Publisher Interview, Project 27). 

Purchasers have primarily included teachers, schools, and districts that have been 
directly involved in field testing. (Summarized from Publisher Interview, Project 
15). 

“The field test touched on Texas, California, New York and Colorado, and a lot of 
the initial sales moved out from there. There were teachers there ready and 
willing to adopt the course and materials.” ( Publisher Interview, Project 20). 

Role of Developer in Marketing 

For many projects, developers played a critical role either directly or indirectly in assisting 
their publishers to identify target markets and the strategies for reaching them. Both developers 
and publishers agreed that IMD developers — given their still forming market niche — played a 
greater role in developing marketing strategies than is typically the case between a publisher and 
developer in the traditional textbook market. 

“[After some preliminary difficulties], NSF then gave us 90 days to find a pub- 
lisher. All of the other programs that had made their initial agreements with 
standard textbook publishers had found that those publishers stayed with them 
until the materials were completed, but then they would eventually walk away 
without signing contracts. We had been seeing what was happening, so we had 
deliberately avoided them... Eventually, we approached one company with the idea 
that [we] would put them back in the marketplace. Up till then, they’d only been 
selling movie films, and had been losing money. I showed them how to go into 
video lessons, laser discs, computers, plus hands-on science materials. They got 
involved in 1987, and the [grades] 3-6 materials turned them around to where they 
started making a profit, and were later willing to invest more in the [K-2] materi- 
als.” (Developer Interview, Project 4). 

“In marketing the materials, it also adds credibility to be associated with presti- 
gious groups. The recognized name and credentials, combined with NSF funding, 
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is stronger than NSF funding by itself.” ( Publisher Interview, Project 4). 

The publisher did not actively seek out or otherwise proactively identify members of the 
target market. Rather, they have relied largely on the training and professional develop- 
ment activities undertaken by the developers themselves to generate interest in such 
alternative curriculum materials. The publisher also relies quite heavily on word of 
mouth referrals between users, and on the fact that NSF’s IMD program is known 
within the math community and people have been awaiting the products funded within it. 
(Summarized from Publisher Interview, Project 7). 

In a few instances and for different reasons, developers doubled as the publishers of their 
own materials. For a small number of projects (less than 10 percent), the decision to self publish 
was a conscious one designed to circumvent the commercial publishing arena in order to make 
the materials available to teachers free or at minimal cost. For another small set of projects 
(again less than 10 percent), the decision to self publish was driven by difficulties encountered in 
finding or keeping a publisher to carry the materials. 

In roughly 20 percent of projects, the developers played a far less active role in marketing, 
choosing instead to leave the marketing to the publishers once development was completed. 

This approach has met with mixed results in those instances where publishers were less experi- 
enced or otherwise hampered in their capacity to carry out their agreed-upon functions. 

Marketing and dissemination were problematic. These functions ended up with a 
group that was just starting in the elementary school market and had neither the 
knowledge nor the enthusiasm to market effectively. In the words of the mar- 
keter, “We did not have a lot of experience in the elementary school market. The 
sales force wasn’t comfortable with the materials and they didn’t put much energy 
into it. They didn’t follow suggestions.” (Summarized from Publisher Interview, 

Project 20). 



Conclusion 

Although the vast majority of the development teams receiving IMD funding produced 
high-quality materials that meet content standards, the development process itself contains the 
seeds of ultimate success or problems. For example, the composition of the development team 
influenced the degree to which materials were judged usable by adopters. More positively, 
field-test sites became early adopters and champions of the materials and the fit with content 
standards was frequently cited by adopters as influencing their decision. 

Few projects collected data related to student outcomes, so there is little evidence besides 
expert judgment that the materials will help students meet high standards. On the other hand, 
few adopters asked for information about student achievement to influence their decision to 
adopt, although parent and teachers concerns during implementation frequently raised questions 
about how well students would perform. 
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CHAPTER IV 



Dissemination 

The relationship between the developer and publisher is a crucial link in the chain from 
development to implementation and use. In many projects, developers and publishers shared a 
vision, resulting in published materials that matched the intention of the developers. However, 
in others, differences between the two led to changes in the product and deviation from the 
original goals. Such deviation, when it occurred, led to problems related to adoption, implemen- 
tation, and use. 

As materials move from developer to publisher, they are considered in the context of the 
market. If a market for reform-oriented materials exists, mainstream or other commercial pub- 
lishers would use their resources to support dissemination because the materials could be profit- 
able. However, the major barrier faced was the perceived absence of a market for reform-ori- 
ented materials. 

Most developers did not systematically identify an appropriate market before proceeding 
with development of materials. Except for those developing extensions of earlier materials, 
developers were more concerned with creating innovative materials than with their eventual use: 

“We were not trying to produce a curriculum that was market driven. Instead, we 
reviewed current middle school curriculums and tried to produce an innovative 
curriculum where one did not exist.” ( Developer Interview, Project 11 ). 

Both developers and publishers repeatedly spoke of a reform-oriented market as compris- 
ing 20 to 30 percent of the total market, although no one was able to provide a reason for that 
number. However, the widespread belief that the market was small meant that few major pub- 
lishers were interested in accepting the materials. Further, even those who published the IMD 
materials cited the need to build the market for reform before significant market penetration 
could be gained. In addition, although developers and publishers believed that professional 
development served an important function in disseminating the materials and acted upon that 
belief, publishers noted their own limited ability to provide the type of professional development 
required. 

Marketing success largely built on positive relationships between developers and publish- 
ers. A further element in success was the activity of the developer in assisting marketing efforts. 

This section begins with a look at developers’ perspectives on why large publishers seem 
reluctant to publish IMD-funded materials. The original intent of the study was to ask major 
publishers about their views of NSF-supported projects and the reasons they were either willing 
or unwilling to publish them. However, major publishers refused to take part in the study, so the 
voices reflected in this section are those of developers only. 

The section then moves to a discussion of the relationships between developers and pub- 
lishers, highlighting elements that support active dissemination and those that seem to be barriers 
to widespread use. The third part focuses on what developers and publishers called “building a 
market for reform” as essential to successful dissemination, adoption, and implementation of 
NSF- funded products. The section concludes with a look at the marketing strategies used and 
notes the importance of professional development as an approach to marketing, but also ac- 
knowledges the limitations in relying on publishers to provide that professional development. 
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Reluctance from Larger Publishers 

There were a few projects where developers signed with larger publishers, either from the 
start, or after first working through a smaller publisher. However, to their chagrin, most develop- 
ers discovered that the majority of the larger publishers were unwilling to publish the materials. 
The developers stated that major commercial publishers consider the reform market to be narrow 
and controversial and tended to shy away from carrying IMD products. Because those publishers 
would not talk with us, we can only report on the developers’ perspective. 

“We’ve been looking to get another publisher, but prospective purchasers are 
seeing the potential controversy. Also, other publishers typically have their own 
line, and they don’t want to cannibalize those sales. ..We came to the realization 
that publishers are very reluctant to take on these new curricula and the expenses 
associated with them. Unless you find one, usually a smaller one, that’s willing to 
take a risk, they’re just not interested... The publishing industry has not been 
known for its innovations. If anything, they’re known for the opposite.” (Devel- 
oper Interview, Project 15). 

Developers felt that getting a publisher is of critical importance, and that they 
were lucky to have found who they did. The PI emphasized that you have to get a 
good publisher, not just a publisher. Many publishers are so large that they don’t 
pay any attention to those doing projects like this. The teacher/developer com- 
mented that she wasn’t sure what would have happened to their project if they 
hadn’t connected with the publisher. She’s not sure they could have gotten the 
materials out. (Summarized from Developer Interview, Project 6). 

“There is still a bottleneck in terms of problems in the publishing field. It is 
unrealistic to expect publishers to participate in killing their own cash cows. The 
standard textbook people are still a problem. The reason [our publisher] is ame- 
nable is because they see educational reform as their publishing niche.” (Devel- 
oper Interview, Project 2). 

“Most curriculum products are published by big publishing companies who can 
afford to give good deals to districts that buy large numbers of books. Big pub- 
lishers are not willing to invest in new curricula, so new curricula are often pub- 
lished by small publishers who cannot afford to offer such deals. This makes 
dissemination of new products difficult.” (Developer Interview, Project 19). 

One publisher in particular identified itself as focusing on the reform-oriented niche of the 
general market. Its focus has both positive and negative aspects for the IMD program. On the 
one hand, the editing and sales staff understand reform-oriented curriculum and can work closely 
with developers. On the other hand, the limited size of the market it pursues constrains the funds 
it can invest in dissemination and staff development. 

Relationships Between Developers and Publishers 

Effective partnerships between the developer and publisher were often characterized by a 
shared vision of the education reform agenda or shared knowledge of important content prin- 
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ciples embodied in the materials, which made the niche market publishers attractive to develop- 
ers. In the best situations, developers and publishers took seriously the need to be full partners in 
promoting and disseminating the materials. On the developer side, this meant that from the 
outset, members of the team realized the importance of finding a publisher who felt comfortable 
in taking the risk to publish niche materials, understood the developer’s vision, and valued the 
need to provide professional development to potential adopters. On the publisher side, the best 
relationships were characterized by a strong commitment to professional development, under- 
standing of the type of material with which they were working, and knowledge of the actual 
content. 

In one successful partnership, the developers were savvy about their relationship with the 
publisher, in part because a member of the development team was familiar with marketing and 
the business aspects of publishing. The developers included contractual provisions related to 
their major concerns: 

“.. .We were careful to write into the contract that we were to participate in all 
aspects of marketing and development. In fact, their sales staff had to be okayed 
by us... We wanted to build a relationship with the publishers and be partners all 
the way...” ( Developer Interview, Project 4). 

The publisher also understood the nature of the materials: 

“It’s clear that this wasn’t competing against textbooks — we had to appeal to 
schools, districts, and states who were interested in looking at fresh approaches to 
improving science education for children. We were competing in a huge market 
but as an entirely new entity.” (Publisher Interview, Project 4). 

In addition to the shared understanding between developer and publisher, successful dis- 
semination of the materials built on the existence of earlier, related materials, which created an 
initial market. 

Another successful partnership involved extensive efforts by the developer to build a 
support network before making the materials commercially available. Building the network was 
possible because the product was fairly well developed prior to receiving IMD funding. At the 
same time, the publisher’s staff spent time with the developers and members of the network so 
they could understand the materials. 

The sales manager and vice president of marketing have gone through the training 
for the materials so they can understand at a deeper level how to represent the 
materials in the marketplace. (Summarized from Publisher Interview, Project 5). 

Less effective developer-publisher relationships were often characterized by a gap in 
understanding between the two parties, which, at its most extreme contributed to a mismatch in 
goals. In such cases, developers were naive in selecting the publisher, and publishers did not 
understand the product, the developer’s vision for it, and the difference between the IMD- 
funded materials and more traditional curricula. The potential for conflict between developer 
and publisher is great because each brings its own perspective to the partnership. Although the 
developers would like the materials to be widely disseminated, their main goal is to have the 
materials implemented in the classroom to advance an approach that they deem important: 
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. we don’t want people buying the materials for the wrong reasons. . .We felt it 
important to argue the integrity of the materials... Since they were into the money 
and we weren’t, we viewed [working closely with the publishers in marketing] as 
counterproductive to our purposes.” ( Developer Interview, Project 7). 

“NSF needs to keep encouraging innovation. The market will do nothing itself to 
encourage the development of better science curricula... Publishers do not care 
about producing innovative curricula but only textbooks, which can be sold for a 
profit.” (Developer Interview, Project 26). 

“...because publishers are commercial, meaning that they have a focus on the 
bottom line, they’re really only willing to work with something that already has a 
market. A commercial enterprise doesn’t have lots of room to create a market. 

To expect publishers to deeply engage in something that steps outside of the 
current market is naive. They simply won’t invest huge amounts for something 
where there’s not yet a substantial market. Educators and publishers have differ- 
ent business practices and goals, and we’re totally at the mercy of the publisher 
and the business decisions they make. NSF used to worry about the product 
sitting on the shelves of the developer, but I tell you, I worry more about it sitting 
on the shelves of a publisher.” ( Developer Interview, Project 10). 

In some cases, developers experienced what they believed were “bad deals” with publish- 
ing houses: 

No member of the development team knew if [the publisher] was offering profes- 
sional development for teachers. One suspected that they did not, and if they did, 

[the publisher] was probably charging extra for it. He also contends that because 
[the publisher] has separate divisions for math, science, and technology, the 
logistics involved in [the publisher] conducting teaching development are likely to 
be difficult. The developers feel that the greatest lesson they learned was the 
difficulty in dealing with a major publisher. The developers have felt powerless 
in getting [the publisher] to respond to their concerns, [the publisher] has pub- 
lished only four modules, which developers feel has turned [the product] from a 
comprehensive curriculum to a supplemental one. The developers are upset 
because they cannot take their materials to another publisher because of the 
agreement they made with [the publisher], (Summarized from Developer Inter- 
view, Project 17). 

The developer said this was a different kind of curriculum — it was for teachers 
not students. It required a new mindset that publishers didn’t have. For example, 
publishers routinely give away copies of the Teacher’s Edition as a marketing 
come on. When they did that with [the product], they were giving away what they 
were trying to sell.... The sales force is not motivated because it’s not that profit- 
able selling books to teachers as opposed to selling books to students. There is 
one teacher to 30 students, so you sell one book when you could sell 30. (Summa- 
rized from Developer Interview, Project 10). 

In other instances, the developers believed that the publishers were simply not ready for the 
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types of products that were being developed with IMD funding. 

These are the most complicated materials that he has ever tried to market. There 
is no formula for selling them and no opportunity for rapid sales. People must be 
trained first. This will never be a runaway best seller. Sales will be slow and 
progressive, and there will only be an incremental increase in market share, with 
lots of work at each level. There are not structures in place to support rapid 
growth. As a result, this type of project is less profitable for them and more risky, 
particularly since they have not control over the professional development side. If 
NSF pulls its support for that, then they Will basically lose their investment — 
that’s the number one lesson he’s learned. (Summarized from Developer Inter- 
view, Project 5). 

According to the project developer, they wanted to find a publisher who was 
willing to market all three pieces... .At that time, publishers were not touching 
software, and distributors of manipulatives tended not to market books. Some 
were willing to take on one piece, but not the rest. (Summarized from Developer 
Interview, Project 6). 

Interview respondents frequently raised the issue of the size and nature of the market that 
might be expected to invest in reform-oriented materials, and the general estimate was that the 
market was between 20 and 30 percent of the total market. Most developers and publishers 
believed that was the market share for which they were competing. However, a few developers 
thought that the market size was underestimated, leading to weak approaches to dissemination: 

“There doesn’t seem to be any overall marketing strategy on the part of [the 
publisher]. Why? They underestimated the demand for these materials in spades. 
They’ve seen them as niche products because the school district has to be ready to 
stretch to do this over choosing a hardbound text, So they only printed 2,000 
copies in the first run. But later on, at one time, they had $2.5 million in back 
orders, and we don’t know how many of these ever got filled. . ..I don’t know if 
the publishers even have a rep just for us... There’s not much devoted attention to 
[the product]. Our stuff is far out compared to the hard cover stuff they carry, but 
the hard cover texts sell better.” (Developer Interview, Project 19). 

While one of the developers is concerned that the “early adopters” have all bought 
the curriculum by now, the publisher disagrees. She says that [the product] had 
“gone beyond the cutting edge and that there are now large scale district adoptions 
taking place, winning people from [a mainstream publisher].” (Summarized from 
Publisher Interview, Project 15). 

For the most part, the publisher’s goal is to generate a profit. 

“We can’t force people to buy our materials, and sometimes I wonder if [the 
developers] realize that.” (Publisher Interview, Project 21). 

The conflict between a developer and the commercial publisher they chose is illustrated by 
a problematic partnership. Features of the materials (Project 24) make it complicated for any 



Final Report on the Evaluation of the N^Tjogl Science Foundation’s Instructional Materials Development Program 



publisher to market because the project involves integrated curriculum, designed to promote inquiry- 
based and cooperative learning. The curriculum is activity- rather than textbook-driven. Given the 
complex nature of the materials, the selection of a commercial publisher with no experience with reform- 
oriented curricula led to clashes between the developer and the publisher. In marketing the materials, 
the publisher used techniques that had been successful with traditional materials. Although the materials 
were designed as a comprehensive set, the publisher turned them into supplementary modules, and only 
published some of them. The publisher justified the decision by arguing that cost factors would influence 
teachers to buy supplemental rather than the comprehensive set. In addition, the publisher promoted 
different modules according to the relevance of the materials for different regions. From the developer’s 
perspective, the marketing approach contradicts the intention of the materials, and almost guarantees 
that they will not have the intended impact. 

Sometimes, publishers were able to convince developers of the need to change their stance. For 
example, a set of materials (Project 10) included only a teacher’s manual. The publisher had great 
difficulty in determining how to sell it because the sales force continued their practice of giving away the 
teacher’s manual to generate interest in the materials. According to the publisher, the lack of a student 
component has implications for the success of the materials. First, there is little fiscal incentive for the 
sales force so the product is less competitive. Second, parents traditionally view student materials as a 
source of information about their children’s lessons. Perhaps in response to both concerns, the new 
edition of the materials will include student workbooks and family worksheets. 

The importance of a full intellectual partnership between developer and publisher is illustrated by 
publishers of two successful supplementary products. Staff at these companies had content knowledge 
that enabled them to identify secondary markets for the products, thus enhancing their profitability. 

In sum, it appeared that effective partnerships between the two tended to be associated with a 
shared vision of the education reform agenda or a shared knowledge of important content principles 
embodied in the materials. 

“Make sure that whoever is developing the product can develop a relationship 
with a publishing group that has not only the monetary resources to mount a good 
marketing program, but who is committed to the product’s underlying principles.” 
(Publisher Interview, Project 4). 

They selected the publisher because they have a lot of respect for their profes- 
sional standards. The staff includes many ex-high school teachers and educators. 

The company also has a strong track record of working with schools and districts 
interested in innovation, which will help them reach their target market. (Summa- 
rized from Developer Interview, Project 21). 

The publisher attributed much of the success of their relationship with the devel- 
oper to their ability to relate as colleagues to each other. This is largely possible 
because it has people on staff who can ‘speak the language’ of the developer, 
since they possess the technical knowledge. Working toward a common goal is 
crucial to the success of a product. (Summarized from Publisher Interview, 

Project 23). 
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“Selling” Reform 



A common refrain from both developers and marketers was that success of the IMD program 
rests on creating a market for reform, which they judged did not currently exist. 

‘Teachers thought the developers were pushing the envelope on changing the way 
mathematics is taught to children, and they said schools and districts were not 
ready for all the parts of the curriculum.” (Publisher Interview, Project 22). 

“High school coordinators are strongly committed to using textbooks and not 
more activity-based materials. Coordinators have a difficult time understanding a 
curriculum which has no textbook.” (Publisher Interview, Project 26). 

“NSF needs to market itself to those teachers who need to change what they do. 

Lots of teachers have the viewpoint that NSF is out there in the ivory tower and 
doesn’t have a clue about life in a classroom. NSF would do [itself] good by 
getting out and promoting themselves as not just ivory tower pointy heads. They 
need to do a little public relations of their own within the teaching population.” 
(Developer Interview, Project 10). 

“Frankly, the majority of teachers out there are not interested in changing* the way 
they teach.. .Districts are looking to us to help change teachers’ attitudes regarding 
teaching methods, and that’s a tall order.. .If there’s any one thing NSF should 
do.. .in the next decade, it is not developing curriculum — though that’s impor- 
tant — [it is] the grassroots retooling of teachers. ..We’ve changed the books, now 
let’s change the teachers.” (Publisher Interview, Project 10). 

Professional development is key to developing a critical mass of teachers who feel comfort- 
able with changed roles and pedagogical approaches engendered by the materials. Publishers 
benefited from staff development as a marketing technique (see below), but they were frustrated 
by their own lack of capacity to provide the professional development necessary to prepare a 
wide variety of teachers to use the materials. For example, the publisher of one of the mathemat- 
ics curricula reported: 

As a publisher, they will do the “up and running” inservice, but they have neither 
the resources nor the skills to do the more in-depth version. (Summarized from 
Publisher Interview, Project 10). 

Another publisher, of a number of the IMD products, said that staff had recently been 
grappling with the distinction between offering training required to support local implementation 
and offering much more extensive professional development that lays the groundwork for chang- 
ing deeply rooted traditional teaching practices. 

The publisher strongly believes that the scope of what is required to adequately 
prepare and equip teachers for doing things differently is far beyond any 
publisher’s capacity or responsibility to carry off. (Summarized from Publisher 

Interview). 
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In addition to using professional development to convince teachers of the need for reform of 
science, mathematics, and technology education, both developers and publishers argued for 
community outreach to develop support for reform. Developers at the high school level were 
particularly concerned about community support for reform. 

“When thinking about reform, NSF needs to look far more broadly than just the 
teachers and the school districts. Parents and communities need to be educated 
and informed about the need for modifications in the curriculum.” ( Developer 
Interview, Project 23). 

The biggest single lesson he learned had to do with the level of public and commu- 
nity education that absolutely has to take place when introducing curricula like the 
one he helped develop. Because the product looks and is so radically different 
from what parents are used to, they experienced much more backlash against the 
product than they had anticipated. ( Summarized from Developer Interview, 

Project 5). 

In the absence of a large market, and without the necessary resources to provide adequate 
professional development and support to teachers to help create the market, some publishers 
focus on more superficial elements of the material in an attempt to sell to the “average” teacher. 
One large publisher believed it was important to package the materials so that they would look 
like traditional texts, even though the content is dramatically different from what is contained in 
those texts. 

The product has lots of change and math reform in it. If you’re trying to make a 
program palatable to the middle ground teacher and you want them to try some- 
thing new, you have to give them some of their comfort factors. ( Publisher 
Interview, Project 15). 



Marketing Strategies 

To build awareness of the materials, all but two products were marketed using some combi- 
nation of in-person seminars, hands-on workshops, booths at trade shows, brochures, presenta- 
tions at professional conferences, catalog mailings, advertisements in journals, information on a 
web site, connecting with professional networks, and direct mailings. Two developers placed 
their materials on the web, and they did not reach a wide audience. Only two actively sought 
approval by state adoption authorities, and they were successful in marketing in the states in 
which they appeared on adoption lists. Others maintained that state adoption processes were 
expensive and conflict ridden, and were unlikely to yield enough to be worth the time and cost of 
pursuing approval. And, there was no effort by developers or publishers to build a market by 
targeting preservice educators. 

Virtually all respondents agreed that in-person awareness seminars or hands-on workshops 
were by far the most effective means of increasing potential users’ understanding and interest in 
the materials. Marketers and developers found that giving such seminars at gatherings, such as 
professional association conferences, attended by members of existing professional networks 
yielded much interest. 

In some cases, less expensive approaches were first attempted with less success, which led 
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the players to conclude that building a customer base requires face-to-face interaction. 

“We found that what works best is an educational approach to marketing. We 
educate people to understand the product, as opposed to just purely promoting it 
and trying to wow them. Then the product sells itself.” (Publisher Interview, 

Project 4 ). 

Direct mailings, videodisk demonstrations, brochures, NCTM conference presen- 
tations and displays, journal advertising, publications, and website postings were 
all used. However, marketers discovered that “you need a person there doing a 
demonstration. To market something at this level and price requires more hand- 
holding, and this caught us off guard.” (Summarized from Publisher Interview, 

Project 1 ). 

A video has been produced promoting the materials along with brochures. But 
the most successful tool in promoting the product has been the hands-on teacher 
workshops. Here teachers get to try out the product for themselves. A marketer 
commented, “It’s like going out to dinner. Would you rather see pictures of the food or 
would you rather be allowed to try it?” ( Summarized from documents and Publisher 
Interview, Project 9). 

However, the high cost of professional development may provide the upper limit for dis- 
semination through staff development. 

“There is no way that we could structure the price of the materials in a way that 
would also support the cost of training... Publishers will sometimes earmark up to 
5 percent of sales for professional support. In our case, that amount wouldn’t 
even come close to what would be necessary to adequately train teachers. For 
instance, if a teacher were going to teach 100 students, the district would spend 
$3,500 [on instructional materials], which would provide $175 for [teacher] 
training. The developers estimate that teachers need at least 2 weeks of training 
during the course of a school year, and it would cost them $1,500 to provide that 
training — and that’s just for one year.” (Publisher Interview, Project 28). 

Summary 

The first sign of deviation from intention to implementation came when developers sought 
publishers, who, in turn sought to sell the materials. Even when publishers and developers 
shared the vision for reform-oriented materials, they faced real problems in the marketplace. 
Both groups talked of facing a small market share and the need to create demand for reform. 
Both also spoke of the limits publishers faced in providing the type of staff development that 
would increase the market share for IMD-funded materials. 

Despite the costs, no other dissemination strategy was as effective as professional develop- 
ment, particularly with educators, including those who had participated in field tests or were 
active in professional networks, who were ready to embrace reform. 
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Adoption 



The IMD program funds the development of materials that meet high content and pedagogi- 
cal standards. A common view is that potential adopters would use similar criteria in selecting the 
materials. Our interviews with adopters indicate that the picture is more complex, and adoption 
decisions are made in many ways, using a variety of criteria. Interviews were conducted with 15 
adopters of supplementary products and 17 adopters of comprehensive products. 

We found that the most successful adoptions were those that engendered teacher investment 
in the materials. In addition, although the criteria used can be seen as elements of a rational ap- 
proach to adoption, in fact, adoption decisions were much more opportunistic — a teacher would 
see materials at a conference and become excited by their possibilities, without analyzing how they 
fit with other school and district priorities; or a marketer would claim that the materials fit the 
standards and a school or district would adopt them, absent an independent assessment. The 
adoption process is also vulnerable to political changes in the district. 

Further, to foreshadow the following chapter on implementation, the process by which 
materials were adopted, along with the substantive criteria used, had an influence on implementa- 
tion and use. Our findings concerning the variety of ways sites approached curriculum adoption, 
and the limitations of the assumption of rationality, provide additional reasons that there are gaps 
between the intentions of IMD products and their actual use. 

Adoption is the point at which control passes from those with whom NSF had direct or 
indirect involvement to those whose actions are independent of NSF influence. Consequently, 
gaps increase, particularly for products that are best implemented in multiple grades through a 
planned process. 

We found adoptions of specific materials clustering geographically. In part, the clusters were 
associated with pilot- or field-test sites, and in part, by the presence of other NSF programs, such 
as Statewide Systemic Initiative projects. In both cases, the original adopters served as models to 
later users. Perhaps increased attention to dissemination efforts that target well-respected indi- 
viduals and districts will yield increased adoption (Rogers, 1962). 

No single approach to adoption is universally related to satisfaction and appropriate imple- 
mentation. However, our study indicates that processes and criteria used to identify and adopt 
curriculum influence the depth of teachers’ investment in its success. In some cases, the degree to 
which an adoption committee represents key stakeholders (teachers, parents, district personnel) 
influenced the amount of acceptance from eventual users. In others, an individual teacher built 
support through successfully modeling use of the product. In still others, adopters’ attention to 
specific criteria, including state and national content standards, state testing programs, and peda- 
gogical strategies affected the way teachers respond to and accept new curricula. 

The section begins with a discussion of the levels at which adoption decisions are made, 
district, school, or individual teacher. We then move to a discussion of the substantive criteria used 
forjudging curricula. The interaction of the process used at whatever level and the criteria influ- 
ence teachers’ investment in the product. 

Levels for Adoption 

The materials we studied were adopted at three different levels: district, school, or class- 
room. At district and school levels, committees were frequently involved in the decision to 
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adopt, although even in those arenas an individual was able to influence the decision. 

Differences existed in the adoption processes for comprehensive and supplementary 
curricula. IMD-funded comprehensive curricula were adopted by district committees, school- 
level processes, and individual teachers. In our sample, fewer than half of the comprehensive 
curricula were adopted at the district level, and the remainder at the teacher or school levels. 
(School adoptions were often led by individual teachers, blurring the line between school and 
teacher adoptions.) About two-thirds of the supplementary curricula were adopted by individual 
teachers. 

District-Level Adoption 

In general, district-level adoptions were led by the district curriculum specialist, but the 
details of the process took many forms. First, although most districts created teams responsible 
for adoption, the teams varied in their composition and scope. For example, the district that 
adopted Project 27 included one teacher from each of the district’s elementary schools; and the 
district that adopted Project 19 had two teachers from each grade level, one principal from each 
school level, and the curriculum coordinator. In some cases district-level staff chose partici- 
pants, and in others, schools nominated their representatives. 

A second difference lay in the scope of the committee’s assignment. For example, the 
committee that selected Project 17 simply looked at the list of state-approved materials and 
selected one that seemed “exciting” to the group. In contrast, a district that adopted Project 1 1 
used a committee that: 1) developed criteria for selecting materials; 2) used the faculty from the 
content area to review the available materials and select a few that met the screens; 3) brought in 
all staff and got their input for the best curriculum to meet the curriculum framework; 4) sought 
parent input; 5) distributed and analyzed rating sheets from teachers, parents, and others; and 6) 
made a recommendation to the Board of Education. 

Although district-level adoptions exemplified quite rational and thoughtful approaches, 
they are the most likely to be affected by political change, as indicated by two adopters: 

“The committee process will be used again, but the adoption method may be 
changed. Currently, the state has its own standards, but districts are allowed to 
choose their own textbooks. The state is now working on an assessment students 
will need to pass if they are to graduate from high school. If this is done, districts 
will have to adopt curriculums which meet the state standards.” (Adopter Inter- 
view, Project 30). 

“The adoption process represented massive reform in the district. . .Next time there 
will be different parents, a different board, and different teachers. . .If these people 
oppose reform, the process could look very different.” (Adopter Interview, 

Project 19). 

In the group of products we examined, 44 percent of the comprehensive products and 8 percent 
of the supplementary products were adopted at the district level. 

School-level Adoption 

School-level adoptions reflected a variety of strategies. Most often, an individual teacher, 
principal, or group of teachers took responsibility for locating a new product in order to better 
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serve the school’s learning goals. Such a process occurred with Project 17, when a principal had been 
seeking an integrated learning experience for the students for a number of years and read about the 
opportunity to work with the project developer as a pilot site. In contrast, Project 5 was adopted at the 
school level, but: 

“At each school, the adoption process has been different. . .The schools that have a 
unified mathematics department do much better, because there is a shared com- 
mitment to using the product. At those schools where the mathematics depart- 
ment is not unified, there can be problems.” (Adopter Interview, Project 5). 

Of the products we studied, 32 percent of the comprehensive and 15 percent of the supple- 
mentary products were adopted at the school level. 

Classroom-level Adoption 

Teachers frequently initiated adoption by bringing the materials into their classrooms. In 
these cases, teachers did not spend their personal funds to buy the materials, but rather they used 
school or district money by recommending the purchase. Initiating teachers heard about the 
materials either through conferences, their professional networks, or because they were involved 
in a pilot or field test. For example, Project 12 was adopted in a number of schools because the 
state professional association presented a workshop on it, and a teacher saw Project 29 at a 
national conference and was impressed by its hands-on nature and links to the community. In 
contrast, Project 20 was brought to a school by a teacher who had been in a field test at another 
location. In both cases, their work gained the respect of colleagues, who asked to use the materi- 
als. The diffusion process reflects the findings of earlier studies of dissemination (Rogers, 1962), 
which points to early adopters as models that stimulate interest in change. 

Teacher-led adoptions comprise 24 percent of the comprehensive and 77 percent of the 
supplementary products. 



Criteria Used in Adoption Decisions 

Whether the product was adopted at the district-, school-, or classroom-level, adopters 
applied multiple criteria in making their decisions. The importance of a particular criterion 
varied in the settings we visited, and we found no sites that used all the criteria in the decision 
making process. Each criterion is discussed in the following section. 

Fit with Standards and State Tests 

Standards were in the forefront of the conversation among committee members and in the 
minds of individual adopters. 

The state standards were the primary consideration for one adopter. He has been 
a key player in the state’s process of defining the content standards, and his 
criteria for selecting materials comes directly from working on state standards. 
(Summarized from Adopter Interview, Project 12). 

The district had been working with experts and consultants to try to develop their 
own “problem-solving” curriculum materials that would meet national standards, 
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which the district had adopted several years ago. This was a very expensive process, 
and they realized they wouldn’t be successful. They were pleased to discover the 
materials that met their needs. (Summarized from Adopter Interview, Project 11 ) 

However, local concerns could supersede the standards; buy-in from the community, especially 
parents, was as important to local success as were the standards; and some teachers were capable of 
sabotaging the implementation process by raising local “scare” issues: 

“Much of the controversy came from the parents of children who were in class- 
rooms where the teachers were not implementing the curriculum well, and stu- 
dents were frustrated. This stemmed from several issues. In some cases, the 
teachers sabotaged the new curriculum because they were not comfortable with it. 

‘I don’t like the curriculum, but I have to use it.’ Other teachers did not under- 
stand the curriculum enough to defend it to parents and students.... Some teachers 
tried to teach it like a textbook, and this didn’t work.” (Adopter Interview, Project 
11 ). • 

According to the co-PI, the biggest single lesson he learned had to do with the 
level of public and community education that absolutely has to take place when 
introducing curricula like [the product]. They didn’t do nearly enough to help lay 
a receptive foundation in the school community for change. Because it looks and 
is so radically different from what parents were used to, they experienced much 
more backlash... than they had anticipated. (Summarized from Developer Inter- 
view, Project 5). 

“When thinking about reform, you need to look far more broadly than just the 
teachers and the school district. Parents and communities need to be educated and 
informed about the need for modifications in the curriculum.” (Developer Inter- 
view, Project 23 ). 

Tests were sometimes mentioned along with standards by adopters. In the best cases, the 
adopters understood how the product helped prepare students for statewide testing. In contrast, 
difficulties arose when standards and tests were not aligned or when the state test changed after 
decisions were made. 

Originally, the teachers believed that the materials would prepare their students 
for the state assessment. However, the assessment changed, and there is some 
concern. Nonetheless, the materials seem to stimulate students so they intend to 
continue with them. (Summarized from Adopter Interview, Project 27). 

By the year 2001, science competencies will appear on the state test, and students 
must pass this test in order to graduate from high school. The staff believe that 
the product will prepare their students adequately for the test. (Summarized from 
Adopter Interview, Project 12). 
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Quality of Teacher Guides and Other Support Materials 

When potential adopters, especially those without strong content background, reviewed materials, 
they often looked at how much the teacher guidebook or other supporting materials could help them use 
the product. Guides were more important when there was limited professional development opportu- 
nity. As noted in the reviews of the content experts, overall, products were weak in either providing the 
necessary guidance to teachers or indicating the extent of professional development required. 

The committee picked materials that teachers could use . . . .These materials included 
teacher videos that demonstrated in a short period how the modules could be taught. 
(Summarized from Adopter Interview, Project 27). 

Availability of Professional Development 

Adopters looked at the extent to which professional development was available and judged 
whether it was adequate to the demands of the materials. They also looked at the expense of 
training, how much was provided by the publisher or developer, and whether there were ongoing 
opportunities to receive assistance. To some extent, the widespread adoption by field-test sites 
was related to the availability of professional development, because participants in field tests 
frequently received free inservice training. 

The teachers participated in the pilot [sic]... three of four of the teachers using the 
materials attended a workshop provided by the publisher and developer. This 
training was three weeks long, and although “there’s always more to learning 
about using materials,” the lead adopter thought this was enough to familiarize 
teachers. (Summarized from Adopter Interview, Project 12). 

The August before they began implementing the materials, the principal, the team 
of teachers who would be implementing them, and several other teachers spent a 
week in training at the developer’s institution... Since they were a pilot site [sic], 
they subsequently had a great deal of contact with the developers, including on- 
site visits. The first year, there were monthly visits, during which the developers 
would observe, troubleshoot, answer questions, and generally just provide teaches 
with support. (Summarized from Adopter Interview, Project 17). 

The original developer conducts eight training sessions each year, sponsored by 
major corporations, and pays teachers for their participation in the training. 

(Summarized from Adopter Interview, Project 2). 

Pedagogy 

At times, adopters sought explicit pedagogical strategies in materials. Adopters mentioned 
student-centered pedagogy and active learning for students as desired methods. 

They chose [the product] because it fits with their philosophy of good pedagogy 
and content. Their philosophy is that students should be given a chance to get 
“their hands on the work and make their own discoveries.” (Summarized from 
Adopter Interview, Project 12). 



Science Foundation 



37 



“It is very inductive, very abstract-random. This suits my teaching style. I like to bring 
in everything and really mix things up. Teachers and students who are creative, non- 
linear (abstract-random) really thrive on the curriculum. The other half don’t do well at 
all with it.” ( Adopter Interview, Project 5 ). 

Student/T< eacher Engagement and Interest 

In general, adopters looked for materials they believed would engage students, particularly 
at the middle and high school levels. The concern that materials capture students’ involvement 
was related to a focus on hands-on, inquiry-based pedagogy, because adopters tended to view 
such pedagogy as enhancing student interest. 

Teachers gave answers that were variations on a theme: namely, that students 
enjoy something different and benefit from a fun, visual approach to mathematical 
concepts. One adopter said, “It is fun. It has the same content as other materials 
but is not as serious. You can look at math from a different viewpoint.” (Summa- 
rized from Adopter Interviews, Project 29). 

There were no well-defined criteria used in the adoption process. However, they 
were looking for materials that would promote students’ retention of content, be 
hands-on to eliminate boredom... (Summarized from Adopter Interview, Project 
1 ). 



Cost 



Particularly in districts and schools in which there were budgetary constraints, adopters 
made judgments about materials based on financial considerations. For example, one district 
moved to a second choice science program because it could put together its own kits rather than 
purchase expensive, pre-packaged kits. Frequently, field-test sites received free professional 
development, which served to attract adopters to those products. Further, teachers who were 
experienced with products served as trainers for others, and received free materials in return. 
The free goods and services stimulated wider adoption within schools and districts beyond the 
field-test teachers. 

“Price and flexibility are very important when deciding on materials. You don’t want to 
spend a lot of money on something that may not work.” (Adopter Interview, Project 
30). 

She convinced the developer to donate 300 copies of the student textbook to the 
science program at her school [as part of her work in field testing]. (Summarized 
from Adopter Interview, Project 18). 

Student Outcomes 

Adopters rarely raised questions about or looked at student outcomes in deciding on materi- 
als to adopt. One example, described in the case study that concludes this chapter, exemplifies 
the type of rational adoption process that included a focus on student learning. In fact, only the 
example we cite and one other setting used information about student learning in choosing 
materials. 
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Adoption of Non-IMD-funded Materials 



Perhaps because the non-IMD-funded materials we compared with the IMD materials were 
comprehensive and drawn from state adoption lists, we found that most sites visited used district 
adoption committees. Further, adopters tended to refer to state standards in their discussion of 
the reasons for the choice. And, while the list of criteria used for IMD and non-IMD products 
are similar, we found some interesting contrasts in the discussions of why materials were se- 
lected. 

Users of IMD-supported products tended to be more concerned about finding challenging, 
engaging materials than did non-IMD users. The following quotations illustrate the lack of 
reform orientation in selecting non-IMD-funded products. 

> 

“What is important in adopting new materials is that they be similar to what 
teachers were using previously.” ( Adopter Interview, non-IMD ) 

“The materials were selected because it was a middle of the road curriculum — not 
too integrated. The actual integrated math was too extreme and teachers were not 
comfortable with it. We piloted the integrated math curriculum for two years and 
had to get rid of it because we did not have math major teachers to teach it. The 
teachers wanted the worksheets for the students to practice the skills and so we 
changed to the current curriculum.” (Adopter Interview, non-IMD). 

In short, adopters of non-IMD materials sought better materials than they had, but their 
interest was in incremental changes that would not place great demands on teachers. 
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An “Ideal” Adoption: A Case Study 



The Product 

The product is a K-6 standards-based mathematics curriculum, which was already avail- 
able when the standards movement took hold in the nation. 

The product was developed by a broad-based team housed at a university. Team mem- 
bers each had a background in mathematics, but there were also specialists with mathematics 
writing backgrounds and teachers with extensive classroom experience. The major developer 
was a nationally prominent mathematician. Two “teachers in residence” were the liaison be- 
tween teachers and the writing team and wrote and edited tasks for the units. Curriculum 
writing involved an extensive cycle of field testing and revision. 

In order to promote the product, and to a lesser extent some other products, the university 
helped to form a corporation, which publishes and disseminates the program. Marketing fo- 
cuses on district-level personnel who are avid for change. The publisher provides support to 
districts and teachers through its professional development group, which sponsors user confer- 
ences, train-the-trainer conferences, and conferences for those teachers who become mentor 
teachers for the program at a fairly low cost. For K-6 adoptions, the publishers provide a set of 
implementation plans and train mentor teachers. 

The District 

The district is an urban school district with a student population composed of 50 percent 
Caucasian, 5 percent Hispanic, and 45 percent African American students. In one elementary 
school that showed high mathematics scores, the enrollment is 85 percent African American. 
The superintendent is African American, as is the mathematics coordinator, who was the driving 
force behind the adoption of a new math curriculum for the elementary grades. 

The Adoption Process 

The product was adopted by a district-level adoption committee. Thirty-five people — 
teachers, parents, and principals — served on the committee. The process was driven by the 
efforts of an energetic mathematics coordinator, who ensured attention to implementation and 
student outcomes. 

The coordinator has a motto about curriculum, “design down and deliver up.” His ap- 
proach is to figure out what students need to exit the system and then find ways to help the 
students master those skills. Working with three teachers, he led a study of exit competencies 
required to turn out “quality workers and producers” at the end of the 12 th grade. Since the state 
test is administered at grades 4, 7, and 1 1, the exit competencies were focused on developing 
benchmarks for grades 3, 6, and 8. 

Once the benchmarks were in place, the coordinator introduced a complex multi-step 
process for selecting the elementary mathematics curriculum. Seven series were introduced 
and piloted — the product was the only NSF-funded curriculum. Each of seven schools piloted 
one product, with two classrooms at each level using the new book and the others serving as a 
control. 

To assess the effectiveness of each series, every week the coordinator sent out unit 
assessment objectives, taken from Bloom’s Taxonomy, asking “where are these in your book?” 
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Only the IMD product and one other series addressed the objectives. In addition, the coordinator 
kept track of students’ performance. At the end of the year, when he plotted the gains made by 
students, he found that no other series approached the effectiveness of the IMD-funded prod- 
uct — it outdistanced all of them by 30 points. The coordinator termed the effects of the curricu- 
lum one of a “different way of doing business in mathematics, one of active learning and using 
strategies.” He was committed to adopting the curriculum, although he realized it would require 
a lot of staff development. 

At the end of the pilot year, the vote for adoption was 32 to 3. Those who voted against it 
believed it would be too much work for teachers. Particular issues raised were that the series 
entailed too much content, required too much set-up time, and had too many booklets for 
students to keep track of. Also, principals were concerned about how to monitor the teachers in 
the implementation process and how to present materials to parents. According to the coordina- 
tor, principals were accustomed to “monitoring by watching students do worksheets,” so he 
designed a principal manual for monitoring implementation through classroom observation. 

To address the issues raised by teachers, the coordinator committed the district to serve 
as the centralized distribution center for classroom and parent-focused materials. When teach- 
ers said the teachers’ guide was Too hard” he called the developers and requested correlations 
for each unit with the national standards and state assessments. Then he was able to tell 
teachers which units had to be “mastered” and which could be “introduced.” He also demon- 
strated model lessons to help teachers see “how it looks in action.” 

The product was introduced simultaneously into all the K-6 classrooms — 18 elementary 
schools — three years ago. When asked why he chose to implement the whole curriculum at 
once, rather than phasing it in, as the developers usually advise, he said: 

“We had to do something right away. These kids were not getting what they 
needed. I couldn’t stand by and watch that, knowing there was a better curricu- 
lum available. That would have been a crime. How could I withhold it from some 
and say, wait, I can’t teach you good mathematics now; maybe three years from 
now. These children deserve all the breaks I can give them.” 

Staff development for implementation was multi-step. The developer provided inservices 
locally, involving all K-6 teachers in a week-long institute to give materials and impart new 
strategies, and assessment tools, which the school district funded. Also, the coordinator was 
familiar with the research on change and sent that research to teachers. He also showed them 
that what they had been teaching was not what was being tested — for example, the emphasis is 
not now on computation, but rather abstract thinking. 

There are signs that the product has been successful. Test scores show that district 
math scores are improving. Fourth grade state test results in mathematics for the elementary 
school that is 85 percent minority are: 

1995: 46.0 1996: 76.9 1997: 64.1 1998: 85.4 

The district average shows similar gains: 



1995:41.0 1996:52.0 1997:53.7 1998:68.7 
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In addition to changes in math scores, teachers also see other signs that students are 
learning math. They see their students applying the strategies they learned in math to situa- 
tions outside the math class. Teachers in the district commented that the product makes learn- 
ing math fun for students and they believe that the use of manipulatives in the activities really 
aided in student learning. 

In selecting a curriculum, the coordinator believes that a district should determine for itself 
what features are important and then design a pilot process that shows how different materials 
can meet the needs of their students. He says that “adoption is a very important process” and 
deserves a great deal of attention. He also believes it is a responsibility of the district to seek out 
the information it needs. In fact, an important component in making the decision to go with the 
product (in addition, of course, to its measured effects) was the excellent service he received. 
When he called with a question, he got an immediate response. “Service sold it,” he said. 
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CHAPTER VI 



Implementation 

In the chain between development and impact, the greatest deviation from what developers 
envisioned occurred during implementation. Problems with implementation were not inherent in 
the products, but rather, related to the extent to which they required changing parents’ and teach- 
ers’ conceptions of mathematics, science, and technology education, and teachers’ ability to 
teach in new ways. However, when well implemented, IMD materials were perceived as having 
positive impacts. 

As noted earlier, we solicited names and addresses of schools using IMD materials from 
the publishers to assess the implementation and impact of the IMD materials. However, publish- 
ers were averse to sharing complete lists of adopters and provided the names of up to 10 
schools or teachers who were using the materials. As a result, we cannot assess the breadth of 
implementation of IMD materials since no universe of adopting schools is available. Further, 
publishers probably did not provide a random sample of schools, and some teachers in the 
schools listed had been involved in the development or field-testing of materials. We selected 
teachers and schools representing rural, urban, and suburban communities from among the 
names we received. 

We conducted focus groups of teachers using the all but three, all supplementary, of the 
products in the study and we observed in 38 classrooms of those teaching comprehensive 
materials. Members of focus groups teachers may not reflect a true cross-section of teachers 
implementing comprehensive curricula because we asked a local contact to invite teachers, and 
in several cases, the teachers were those most familiar with the materials. Therefore, generaliza- 
tions cannot be made from the teachers and classrooms visited. The sample’s bias toward 
“good” sites underscores the difficulties of implementation because if such sites have problems, 
less favorable settings are likely to have more. We believe the focus groups and classroom 
observations provide valuable information about teachers’ assessments of materials, their use in 
classrooms, the factors influencing implementation, and the perceived impact on teachers and 
students. 



Content 

Teachers using the materials in their classrooms rated the content of the materials similarly 
to the Expert Panel, but teachers expressed some concerns about both science and mathematics. 

Teachers using the mathematics curricula generally praised how well mathematics content 
was connected to national standards. In addition, teachers were complimentary about the use of 
manipulatives and how well mathematics and technology were integrated. Teachers gave one 
comprehensive elementary school mathematics curriculum highest marks because “It was the 
first one that made a change in mathematics [in the school].” 

At the same time that teachers praised the content covered in the IMD mathematics mate- 
rials, they also expressed some concern. Two comprehensive curricula (one at the elementary 
level and one at the middle school level) were criticized for not presenting enough basic math- 
ematics facts, while one high school curriculum received mixed reviews from teachers (some 
positive and some negative) for its use of graphing calculators. In addition, teachers of one very 
highly rated elementary school mathematics curriculum cautioned others about its sequencing. 
Because the curriculum so transformed the teaching of mathematics, it was difficult for students 
who encountered it for the first time in the upper elementary grades and for students who trans- 
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ferred into the school at mid-year. 

Teachers using the IMD science materials also generally gave them high marks, includ- 
ing their linkages with state and national standards, their investigatory approaches, and their 
conceptual base. Occasionally, an interdisciplinary curriculum at the middle or high school level 
would be criticized for not having enough mathematics content, but this was rare. The single 
largest complaint about elementary science materials lay not with the content or the theory behind 
the content, but rather with keeping track of the contents of science kits provided. Someone had 
to ensure that the kits were complete so that another classroom of students could use them; for one 
product, the district shipped the materials back to the publisher to refill and then return to the 
school. 

Teachers also expressed some concern about readability of materials and use of technology, 
which became barriers to implementation. 

Implementing IMD Materials as Intended 

The extent of implementation varied between comprehensive and supplemental materials. 
More comprehensive materials were adopted district-wide than were supplementary materials, 
requiring that all teachers implement the materials. In these cases, use of the materials and the 
extent of implementation varied, usually depending upon the teachers’ skills and experience and 
the extent to which teachers were able to engage students. Teachers often needed to move from 
a didactic to a more investigative teaching pedagogy for both the math and science materials. If 
teachers were inexperienced, had received little staff development, or were reluctant to change 
their teaching methods, the materials were less well implemented. 

Because supplemental materials were often chosen by the teachers who used them, they 
were used as enrichment and implemented as they were intended. These supplemental materials 
included videos, interactive software programs, and manipulatives of various types. However, 
even teachers who chose products were unaware of materials that enhanced their value. For 
example, none of the three teachers interviewed knew that a set of curricular materials accom- 
panied Project 6. Similarly, a middle school and high school science supplement also came with 
curriculum materials at various grade levels, but the schools visited were aware of only one of 
the half dozen available kits. 



Factors Facilitating Implementation 

Multiple factors facilitate the implementation of IMD materials. This discussion focuses on 
the comprehensive mathematics and science materials, as they require the most supports. 

Among the factors that are often critically important are: 

• Extensive and sustained professional development for teachers; 

• District and school level support, including visible advocates for the materials; 

• High teacher ratings on the quality of the materials; and 

• Supportive educational technology. 

Professional Development 

Training and support for teachers were crucial to successful implementation of most of the 
comprehensive mathematics and science curricula. Support mechanisms include initial and ongoing 
formal training, in-class supports, product networks, and mentor teachers. 
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Districts and schools implementing comprehensive mathematics and science curricula found 
professional development to be an essential component of successful implementation. In fact, many 
teachers and district coordinators believed that it would be impossible to implement these curricula 
without training. While training involved both pedagogy and content, those sites implementing math- 
ematics curricula placed more emphasis on the need for pedagogical training than did those implement- 
ing science products. In interviews, science teachers noted that the field always had a hands-on ap- 
proach, while mathematics teachers reported less experience in the pedagogy required by IMD materi- 
als. 

“The change from traditional instructional techniques to those required by [the 
project] is drastic. It takes an average of four years to completely change the way 
instruction is provided. It is like being right-handed and being forced to become 
left-handed; it’s very clumsy and awkward at the beginning, and takes a long time 
to become proficient.” ( Focus Group, Project 5). 

The amount of professional development for comprehensive curricula varied considerably, 
with most projects receiving somewhere between one and three weeks of initial training. Train- 
ing was often conducted by developers during summer institutes, with occasional follow up. The 
prominent role of developers in professional development may be related to the fact that so many 
of the sites we visited were involved in field tests of the materials. 

Teachers noted that while initial staff development is important, it has to be ongoing as 
teachers use the curriculum throughout the year. 

“Staff development is an essential springboard for using [the product]. It has to 
be ongoing as you teach through the first year. You need to focus on the math 
content in each unit and you need staff development to do that. It’s not just unit 
training, however, but learning to understand mathematical concepts that span the 
units.” ( Focus Group, Project 19). 

Teachers responded positively to training in which the materials were modeled for them, 
they learned just as their students would be learning, and they practiced teaching the model to 
other participants in the training. Such professional development opportunities led teachers to 
become engaged and excited about working with students using the materials. 

District support for professional development was a key factor in some sites, particularly in 
large districts where the curriculum was the curriculum in mathematics or science district wide. 

In addition to support from the developer, the district implementing Project 27 has a Professional 
Development Center that sent out teacher trainers to help classroom teachers with implementa- 
tion. This site has also had the materials in place for a number of years and has built up a cadre 
of experienced teachers who now act as teacher consultants to others as they implement the 
product. 

Because supplements are often adopted by individual teachers rather than by schools and 
districts and their use is optional, teachers tended to receive considerably less professional 
development unless they had been involved in the pilot or field test. More often, teachers had 
seen a demonstration or attended a brief workshop on the materials or received no training at all. 
In several instances, a single teacher had attended a workshop and then conducted a workshop 
for other teachers in the district (e.g. Projects 6 and 14). Teachers using several other supple- 
ments received support through other kinds of networks. For example, Project 20 is tied to a 
professional association, which regularly scheduled dinners at which participants could share 
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their experiences using the product, thereby building and supporting a network of users. 

In cases in which teachers began using materials before they received training, school- 
based mentor teachers proved to be invaluable resources. 

Projects 2 arid 5 had technical support numbers or web sites that offer assistance for sites 
and/or networking opportunities for teachers. However, use was low. One teacher thought it 
would not be useful because of ‘the unreviewed garbage’ that would be on the web. 

District and School Support 

Leadership at the top can make an important difference in the ease with which curricula are 
implemented at the school level. The way to acceptance of new curricula is smoothed by articu- 
lating the importance of the change, building support from teachers, parents, and the community, 
and providing resources. Such actions can bring about the investment in success similar to that 
associated with successful adoption processes. Examples of strong district leadership include the 
district mathematics coordinator in the case study above, who carefully built support from teach- 
ers for the curriculum, then backed up the district’s decision to adopt it as the mathematics 
curriculum for all elementary schools by providing extensive professional development and 
teacher assistance. 

Leadership can overcome major barriers to implementation. For example, Project 5 seems 
to require a visionary to convince teachers, parents, and the community that it provides high 
value to students because it is so different from traditional mathematics texts and entails such 
extensive staff development. Teachers in the site we visited received 100 hours of training and 
support, costing the district $10,000. In a school-wide adoption, the principal of the school 
implementing Project 17 provided block scheduling time to support implementation. 

Centralization of some functions at the district level was also an important support for 
elementary science curricula, many of which include kits of consumable materials that require 
replenishing. By taking responsibility for maintaining these kits, districts increased the ease of 
use for teachers, who did not have to spend time scrounging for materials. 

In contrast, the experiences of an urban district that tried to implement Project 17 provides 
an example of what can happen when these features are not in place. The district decided to 
adopt this entirely technology-based curriculum. Most schools in the district, however, did not 
have the appropriate infrastructure, and the district was not forthcoming with additional funds for 
computers. And, although the district provided staff development, the amount was insufficient to 
teachers’ needs because they were required to learn the technology, the content of the curricu- 
lum, and new pedagogical approaches. According to the director of science for the school 
district, the district has not been able to keep up with the training needs of teachers. Although 
teachers like the curriculum, it has not been widely used. 

Features of the Materials 

Implementation was enhanced when teachers were positively impressed by the materials. 

In general, teachers of NSF curricula in both mathematics and science rated the products they 
were using very highly. In all but a few instances, teachers thought the materials were some of 
the best they had ever seen or used. Features that particularly appealed to them were: ease of 
use, an excellent teachers guide, activities that they believed fostered improved student engage- 
ment and student learning, and the materials’ adaptability for students with diverse abilities and 
learning styles. 
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Educational Technology 

Technology was more prominent in the supplemental materials than in the comprehensive cur- 
ricula. The three main types of technology are computer programs, CDs, and videos. Both computer 
simulations and videos help teachers to demonstrate concepts in visual form that are difficult to explain, 
especially concepts with a high level of abstraction. 

“The hands-on activities and simulations successfully help students to see how 
what happens at the microscopic level drives what is observed at a macroscopic 
level. Unless a teacher is really good at describing this relationship, helping 
students to make that connection can be very, very tough to do, particularly for 
students who are not ready for abstract thinking.” (Focus Group, Project 16) 

Technology also functions as an important tool to save time and take the tedium out of 
collecting, recording, and graphing data. It allows students to see their results immediately and 
leaves more time for discovery and analysis activities. 

“Because of this technology, students are spending less time doing the grunt work 
of collecting and recording data, and more time on analysis, that is, higher order 
thinking.” ( Focus Group, Project 17). 

Barriers to Implementation 

The barriers to implementation include the reverse of the facilitating factors, as well as 
resistance to the type of curricula represented by the IMD-funded materials. Barriers include: 

• Lack of skills and knowledge by teachers, especially for those curricula that 
include reform-based pedagogy and new (often interdisciplinary) content; 

• Active resistance to the curriculum, especially by school staff and parents; 

• Lack of alignment between the materials, district curriculum standards, and norm- 
referenced tests; 

• Absence of supportive technology; and 

• Lack of other significant resources. 

Teacher Skills and Knowledge 

As noted above, the IMD materials require teachers to adopt reform-oriented pedagogy that 
is far more student-centered than teacher-centered, more discovery and investigative than direct 
instruction, and more conceptual than rote. Some teachers adapt to these role changes, while 
others find the transition far more difficult. As a teacher reported about an IMD-funded high 
school program: 

“Teaching [the materials] requires a special personality and a special drive and a 
special person. Someone who picks up the newspaper every night and reads it. 
Someone who watches TV every night and knows what’s going on in the world. 

Someone who is willing to go out on a limb and have an opinion even though they 
might get knocked around, but at the same time someone who doesn’t have all 
the answers. You can’t have the right answer because there are no right answers 
for most things. You have to present both sides of the argument. You’ve got to be 
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able to let your hair down. Have fun with the kids. Oh, but there are teachers 
who will stand up and say, ‘This is the way.”’ ( Focus Group, Project 2). 

Very often it appeared that the materials changed role expectations in ways teachers found 
difficult 



“[The product] encourages independent thinking and decision making. Answers are not 
given. But for teachers this can be very difficult to digest. A major lesson learned is 
that it is difficult to change the thinking of teachers. Teachers are not used to students 
getting the answers for themselves.” (Developer Interview, Project 4). 

“[The product] requires teachers to give up control and let students take the lead. The 
materials can’t simply be lectured. The teacher’s materials are very extensive — giving 
very detailed lesson plans and ideas of what to do with problems — but more often than 
not, the teacher’s materials don’t give the answers to the questions because there are six 
different ways that students could answer the problem, and many teachers at first find 
this disconcerting.” (Developer Interview, Project 5). 

“In order for these materials to be really successful, there needs to be some corre- 
sponding change in teacher strategies. The teacher becomes more of a facilitator, 
rather than being the person who has all of the right answers. There is lots of 
group work involved in the materials, and the teacher really has to direct the 
learning. Questions tend to be open ended. There is a low level of interpretation 
involved in analyzing data sets for there really isn’t one right answer to any 
question posed. A teacher needs to be prepared for the fact that there are multiple 
ways of looking at the data set, and students will often come up with very differ- 
ent answers, which can be unsettling for some teachers.” (Adopter Interview, 

Project 30). 

“New curricula cannot be implemented without knowledgeable teachers who 
understand how science research is done. Most middle school science teachers do 
not have a strong science background and will not gain one in a five-day work- 
shop. A major lesson learned is that teachers do not easily change their mindset 
about how they should teach. Reform curricula are student centered as opposed to 
teacher centered, and the role of the teacher is to facilitate. Many teachers cannot 
change their style.” (Developer/Publisher Interview, Project 12). 

“Teachers need to know the materials extremely well to do a lesson. You cannot 
just open the teacher’s manual the day of an assignment. You need to read ahead 
in the lessons because it takes time to understand the program.” (Focus Group, 

Project 10). 

In districts that mandated a comprehensive IMD curriculum without sufficient professional 
development and other support, some traditional teachers refused to implement it, and others 
have implemented only those features most consonant with direct instruction. 
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Active Resistance to the Curricula 



The introduction of NSF materials, particularly in mathematics, created considerable 
controversy in many sites implementing comprehensive curricula. Because decisions to use 
supplemental materials are typically made at the teacher level and do not represent a single 
approach to teaching the subject matter, their use has raised little resistance. 

Resistance occurred on multiple levels. School board and community opposition to Project 
1 1, for instance, was clear when the materials were adopted as the sole curriculum for the district 
by the narrowest of margins in the school board vote. When the materials next came up for adop- 
tion, a strong back-to-basics constituency group on the board led to a considerable struggle to 
keep the curriculum in place. Teacher implementation was inconsistent as well. According to the 
district curriculum coordinator, some teachers opposed the curriculum and tried to sabotage it. 
Others did not have the deep content knowledge or pedagogical skills to implement it well. Par- 
ents of students in those teachers’ classes are, understandably, opposed to the materials. 

In some districts, either for philosophical reasons or due to teacher, community, and/or 
parent resistance to the IMD products, the materials were adopted as an “alternative” curricu- 
lum, as was true for Projects 17 and 23. In one school implementing Project 17, the principal 
is its major advocate, and the team implementing the program has received considerable public 
recognition, causing deep resentment among other teachers who believe the implementing team 
gets preferential treatment. 

Parent resistance to IMD materials arises from concern about content and the lack of 
homework. Parents in the district implementing Project 19, for instance, believe that computa- 
tion is not adequately covered. Many also expected and wanted to see their children bring 
home more traditional homework assignments. In addition to worries about homework, parents 
in some communities raised questions about their children’s preparation for the next level of 
schooling. Some parents think their children are more challenged by Project 17, but a substan- 
tial number also equate learning with memorization. Such parents believe that students require 
more “rigorous” (i.e., traditional) mathematics in order to be prepared for high school. 

When parental concerns were addressed, the most common way of doing so was with 
ongoing conversations. As one teacher reported: 

“I spend a lot of time in the evenings talking to parents, and both before and after 
school talking to students. I have to work with parents and students to demon- 
strate to them that they are learning.” (Focus Group, Project 5). 

In addition, the district adopting Project 19 created a set of materials emphasizing compu- 
tation that could be used as homework. Districts implementing Project 11 (mathematics) and 
Project 12 (science) also expressed the desire to strengthen connections to students’ homes to 
overcome parental opposition. 

Lack of Alignment Among the Materials, District Curriculum Standards, and Norm-referenced 
Tests 



Both the Expert Panel and teachers agreed that the IMD materials were closely aligned with 
national standards in mathematics and science. However, some users told us the products were 
not aligned with district standards or with the norm-referenced tests that states and districts con- 
tinue to use to measure student achievement. Dissatisfaction with the materials developed when 
these district standards and tests played a major role in the school district. As one teacher noted: 
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“If standardized testing is fact-based, the curriculum needs to include more facts, not 
vague themes.” (Focus Group, Project 15 ). 

Until assessment and district-developed standards catch up with reform-focused content 
and pedagogy, IMD materials will find tough sledding in these school districts. 

Absence of Supportive Educational Technology 

Although technology can facilitate implementation, it can also serve as a barrier. Some 
teachers found that materials such as videos and computer software were dated when they 
began using them. Teachers using Project 1 , for example, found some of the early computer 
software products to be archaic and unsophisticated, although more recent offerings from the 
developer were much improved. Teachers implementing a CD-ROM program (Project 12) 
found it was not particularly user friendly. 

Most sites using computers cited hardware problems as an ongoing barrier to usage as 
well. Many sites had an inadequate number of computers in the classroom, or had to schedule 
time in computer labs. This problem was found among both low- and high-wealth districts. In 
addition, maintaining hardware is a universal problem because either teachers or their students 
must do repairs when the machines malfunctioned. 

Lack of Resources 

A number of schools struggled with other resource shortages. Some schools were lacking in 
basic equipment and supplies to conduct laboratories. Many schools had no water source in the 
classroom, which presented problems for many laboratory-based projects. Students often carried 
water into the classroom in buckets. Others struggled with more basic shortages, such as not 
having enough books to go around for their students. 

Implementation of Non-IMD Materials 

In order to provide comparisons with the IMD-funded materials, we also studied materials 
that did not receive IMD funding. All the non-IMD materials selected were comprehensive 
curricula. Further, we selected the materials from state adoption lists from states that have 
curriculum frameworks or standards in place, increasing the likelihood that the products would 
hold similar intentions to those funded by NSF. We conducted focus groups and class observa- 
tions of teachers using a wide range of curricula that were fairly evenly distributed from tradi- 
tionally-oriented texts (e.g., content is presented, followed by class exercises and homework), 
to those described as “traditional with a few hands-on activities,” to products more similar to 
IMD-funded materials. Comments made by focus group participants make it clear that some 
commercial publishers represented in the comparison sample have made efforts to adapt their 
products to respond to national curriculum frameworks and standards. Staff informally rated the 
materials as more or less reform oriented, using the framework the Expert Panel used to evaluate 
the materials. 

Teachers in sites using non-IMD-funded curricula were aware of both national curriculum 
frameworks and standards and the trend towards student-centered, discovery oriented approaches 
to teaching. Some districts using more traditional curricula had adopted them in the early 1990s, 
and teachers indicated that their districts either had already adopted or would soon be adopting 
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new curricula incorporating a greater emphasis on discovery-oriented, hands-on approaches to 
instruction. One district expects to adopt an IMD-funded curriculum, while others mentioned non- 
IMD-funded curricula developed by large commercial publishers. One teacher commented on the 
impact that the movement in the professional community has had on his perception of their school’s 
text: 



When we originally chose this text, we thought it worthy of a 5 (i.e., on a scale of 
1-5), but now I’d rate it only a 2. The needs of students have changed, and the 
material is disconnected from real life situations. The world is data-oriented, and 
this book does not address that. (Focus Group, high school mathematics cur- 
riculum). 

Overall, most teachers in the focus groups of non-IMD users were at least moderately 
satisfied with the curricula their districts are using, with those using more traditional texts tending to 
be less satisfied. Most curricula were used as intended. However, one middle school science 
curriculum designed to be comprehensive was used as a supplement in one district we visited. 
Teachers in that district believed the reading level was too difficult for their students, reflecting an 
issue that was also raised about IMD materials. 

Teachers using more traditional materials praised content coverage, ease of use, and the 
plentiful practice worksheets that accompany texts. Not surprisingly, focus group participants also 
reported that teachers in their district feel quite comfortable teaching with these curricula. Criti- 
cisms of traditionally oriented texts focused on: inadequate concept development, a lack of variety 
in instructional techniques, insufficient number of hands-on activities, and failure to make real- 
world connections for students. Some teachers supplemented the texts with materials intended to 
address these weaknesses (e.g., teachers in one school supplement their mathematics instruction 
with an IMD product). 

In contrast, praise and criticism of more reform-oriented curricula were similar to that for 
IMD-developed curricula. Teachers praised them for their: development of concepts, pedagogical 
approaches, real-world applications, and their capacity to engage students. One teacher, for 
instance, commented of her curriculum: 

The manipulatives are very good. At all levels, students need some type of hands- 
on activity. You can’t always just work off a ditto sheet — adding, subtracting, and 
what not. The manipulatives make them think and then they have to find a reason 
why they got their answer. Then they can show it to you and their classmates as 
well. (Focus Group, non-IMD elementary mathematics curriculum). 

Criticisms tended to focus on insufficient content coverage, lack of drill and practice exer- 
cises so that students can attain mastery of concepts, the amount of preparation time the materials 
require, and the fact that the materials require tremendous change in teacher practice. 

Non-IMD-funded materials faced barriers to implementation similar to those faced by IMD- 
funded materials. In many ways this is not surprising because both the facilitators and barriers 
generally arose not from the materials, but from school and district policies and practices that 
affect curriculum use. Users of non-IMD materials mentioned the following as facilitating or 
serving as barriers to use: 
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• Professional development 

• Adequacy of resources 

• Teacher skills, knowledge, and attitude 

• Alignment among materials, district curriculum standards, and norm- 
referenced tests 

Professional Development 

Opportunities for professional development were important to the implementation of the 
non-IMD materials, although in most instances teachers reported receiving less intensive profes- 
sional development than did teachers using IMD-funded comprehensive curricula. Many of the 
IMD-funded districts and schools had been pilot-or field-test sites, and professional develop- 
ment was often conducted by the developer, whereas non-IMD funded professional develop- 
ment, if received at all, was either provided by the publisher or the district. Training for non- 
IMD-funded materials was more often geared toward covering the content of the curriculum or 
state and local frameworks than toward pedagogy. The non-IMD materials required less change 
in classroom practice, overall, than did the IMD materials, and the amount and type of inservice 
may be related to the nature of the materials. 

Insufficient professional development was a barrier to implementation of one of the more 
reform-oriented non-IMD-funded curricula. In one district, issues arose similar to those that 
affected the implementation of IMD-funded materials: 

The company supplied the district with some staff development, but because the 
company was small, they could not offer development for all teachers in the district. 

The district had to do inservice training on its own. . . .The district made the commit- 
ment to do some inservice for some leadership teachers, but it didn’t train many 
teachers. The next year we went into the adoption of Language Arts so that took 
priority. That’s the way it is . . .The one-shot deal does not work because the 
teachers were lost. It was a dramatic change for them. . . .We’ve had a lot of new 
teachers come in who were given the curriculum and not trained. . ..We probably 
need more inservice on it for us ourselves to understand what we’re doing. It’s 
completely different from the way things were done. [We need] more time in the 
classroom for the students to process this, too. It is demanding of the time and 
sometimes very difficult to serve justice to it. (Summarized from Teacher Inter- 
views, non-IMD elementary mathematics curriculum). 

One teacher in a district that is contemplating a change to an IMD-funded middle school 
mathematics program was apprehensive about the shift because of the intensive staff develop- 
ment she anticipated the district would be required to provide in order to implement it well. 

High school teachers reported receiving less staff development than did middle school and 
elementary school teachers; in some instances teachers received no professional development at 
all. Teachers in one school seemed unfazed by this lack of professional development, comment- 
ing that high school teachers in their district typically did not receive professional development. 

A collegial work environment was mentioned as being important in two sites. In one site, 
the teachers frequently meet on an informal basis and share ideas about what works in their 
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classrooms. The three teachers in another school receive district support through a common planning 
period, which gives them ample opportunity to share ideas about how to use and supplement their texts. 

Adequacy of Resources 

As was true for IMD-funded curricula, the presence or absence of adequate resources 
made a crucial difference in how well the materials were implemented. In one district, the 
school board assumed district-wide responsibility for replenishing consumables. The adequacy 
of the teacher support materials that accompanied the curricula was also important to successful 
implementation. For example, a high school mathematics curriculum was complimented for 
including many teaching tools: 

It comes with very good examples — two or three examples and plenty of practice 
problems. Everything is clearly defined with lots of explanation. They provide us 
with a lot of extra teaching tools, which are handy — transparences for instruc- 
tional purposes, a manual for testing, a solution manual, a practice book with 
additional problems for students to practice, and alternate workbook. (Teacher 
Interview, non-IMD high school mathematics curriculum). 

On the other hand, implementation suffered when resources were not available: 

We don’t have enough books, and therefore a set of books are shared between 
classes. The teachers have to juggle the books around when two teachers are 
teaching the same unit. There is no ownership of books since we don’t have class 
sets so the kids trash them, tearing pages. . ..The kids can’t take the books home 
and study at night. (Teacher Interview, middle school non-IMD science curricu- 
lum). 

Teacher Skills, Knowledge, and Attitude 

The districts using non-IMD reform-oriented curricula reported more instances of implemen- 
tation being affected by teacher skills, knowledge, and attitudes than did those using more tradi- 
tional curricula. One district using an integrated mathematics curriculum in its high schools had 
pilot-tested another integrated curriculum, but found the mathematics teachers did not have a 
strong enough background to teach the alternative curricula. In another district, some teachers 
resisted a more reform-oriented elementary mathematics curriculum because of the tremendous 
change in pedagogy it required. A teacher using an integrated high school mathematics curricula 
commented that the main complaint she hears is “a reluctance to give up a certain amount of 
material, and the integrated curriculum is seen as almost requiring that you do a certain amount of 
that. And there is still an impression that when you go with the integrated curriculum, they don’t 
give you as much of practice in drills and skills.” 

One teacher expressed her belief that teacher attitude can have both a positive and negative 
impact on student learning: 

I think it has to do with the attitude of the teachers, too. If the teacher is having a 
hard time with it (the curriculum) and being negative about it, it kind of reflects 
also on student learning. But if the teacher is really excited, motivated to do it, 
and feeling good about it, the students will, too. This new way of teaching. ..has 
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allowed people to see math in a positive way — it’s not so threatening. 

Alignment Among the Materials, District Curriculum Standards, and Norm- referenced Tests 

Teachers, even when they liked the more reform-oriented curricula, were critical of incon- 
sistencies among district curriculum standards, the curriculum they use, and the standardized 
tests that their students are required to take: 

For many teachers, there continues to be a certain sense where the criteria of 
success is how many students can number crunch effectively. That is still a strong 
criterion by which they judge the success of a program. “In a sense, we want it 
both ways. We like to have higher order thinking. We like this idea of an inte- 
grated curriculum. We like all these things, but we are very hesitant to pursue 
that at the expense of students not maintaining the level of skill and effectiveness 
at working with equations ...” “Teachers are going to tend to be pulled toward 
maintaining testable skills because that is the assessment tool that will determine 
whether they are doing their job correctly. And so it tends to be, ‘How effective 
am I in being able to get the right answer.’” (Summarized from Focus Group, 
non-IMD high school mathematics curriculum). 

This mismatch may drive many of the comments voiced by teachers about the non-IMD- 
funded projects (and some IMD-funded projects) when they complain that the materials pay 
insufficient attention to mastery of skills. 



Summary 

Overall, successful implementation relies less on the materials than on local factors, the 
existence of appropriate professional development and support, and strategies that gather 
support for the product from teachers and parents. 

When teachers did not receive appropriate professional development, they implemented 
materials in ways resembling traditional practices. In other settings, where district- or school- 
level support was great, or where teacher enthusiasm had been developed through the process 
used to make adoption decisions, implementation was more faithful to developers’ intent. 

A similar problem exists in terms of community support. When parents are not persuaded 
of the value of reform-oriented approaches to curriculum and instruction, they present barriers 
to implementation. They do so by such actions as pressuring teachers and administrators for the 
type of homework that “looks like” mathematics or science as they knew it. In contrast, com- 
munities that work closely with parents experience fewer implementation problems stemming 
from their objections. 
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CHAPTER VH 



Impact 

The evaluation of the IMD materials addressed impact in three ways. First, we were 
interested in how those who were using the materials, either IMD-funded or non-IMD, assessed 
the impact. Second, to the extent possible, we sought information about how materials affected 
student learning. The study did not include testing of students; rather, we requested information 
during site visits about data collected at the classroom, school, or district level. 

Finally, the study focused on how the use of materials affected classroom practice. Data 
related to classroom practice came from two sources: interviews or focus groups with users of 
the materials, both IMD and non-IMD; and observations conducted by WestEd and Abt Asso- 
ciates Inc. staff, using a structured instrument (see Appendix 3). 

The interviews, focus groups, and observations that provided the information for this 
chapter were conducted in a variety of settings, including urban, rural, and suburban schools, 
and schools in which adoption was fairly new and those that had longer experience with the 
materials. 



IMD Materials 

Users of the IMD materials assessed their impact in multiple ways, including the extent to which: 

• Teachers believed that the materials increased their own content knowledge; 

• Teachers believed the products increased their use of reform-oriented pedagogy; 

• Products increased student engagement; and 

• Products increased student achievement. 

Content Knowledge 

Teachers of IMD-funded science products were more likely to report that using the mate- 
rials increased their own knowledge of content than were those who used the mathematics 
materials, perhaps reflecting differences in how well they were originally prepared to teach the 
subject. Elementary school teachers appreciated the theoretical base that the NSF materials 
included, and high school teachers who used Project 5 believed their knowledge was extended 
through using the materials. 

Teachers who used the science materials, particularly in elementary schools, reported that the 
IMD-funded materials increased their knowledge of science. Elementary school teachers often 
have little background in science, so schools often looked for strong teachers’ manuals with clear 
concepts of what to cover. The teachers in the focus group for a comprehensive elementary 
school science project highlighted their appreciation for the clearly presented new content: 

For teachers who do not have a background in science, the materials give them a 
structure and a story line, since each concept and each grade builds upon the next. 

It provides a good introduction to certain meaningful science concepts for teachers. 

It becomes a base for teachers to use to teach science. The first-grade teacher in 
the group said that the curriculum gave her greater comfort with science: “It made 
me not dread science.” (Summarized from Focus Group, Project 21). 
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Similarly, teachers reported that the manual that accompanied Project 27 was clear and 
easy to follow, with a thorough overview and presentation of kits. Even supplemental materials, 
such as Project 20, were cited as improving understanding of science concepts and making it 
easier to teach science. 

In contrast, teachers also reported that some IMD science materials required prior content 
knowledge. One teacher using a high school course commented: 

“I think the materials would scare the crap out of any teacher who is weak in 
science. You can’t go to the text to educate yourself. You have to go to the 
research, which can be intimidating.” ( Teacher Interview, Project 16). 

Reform-oriented Pedagogy: Teachers’ Views 

An explicit purpose of the IMD program is to encourage reform-oriented pedagogy by the 
nature of the products it supports. Teachers of the comprehensive IMD materials often remarked 
that the products stimulated them to use more student-led investigations and discovery activities 
for students, hands-on exercises, and exploration of mathematics and science concepts. When 
teachers of IMD materials remarked that their teaching had not changed, it was typically because 
they reported that they were already using an investigative, hands-on approach. 

With the mathematics curricula, most teachers at each level believed they had changed the 
ways they taught, especially in moving away from teaching by rote. 

“The classroom is more active and student driven. The teacher’s role is different. 

You are a member of the community and not just into exposition. You can’t 
predict what’s going to happen in the lesson because students often have ques- 
tions that the teacher can’t anticipate. It creates a learning community in the 
classroom, he content is so different that it drives changes in practice.” ( Focus Group, 

Project 19). 

“We’re not teaching by rote but by solving problems, transmitting data, and working in 
teams.” ( Focus Group, Project 8). 

“It has changed the way I teach dramatically. It has changed how I teach, how I 
deliver information, how I engage students to be active participants in their 
education. It has also changed the physical appearance of the classroom (it’s 
messier now), and how I assess student learning. It is exciting to watch kids share 
knowledge, and to share their excitement in the learning process.” (Focus Group, 

Project 5). 

With the comprehensive IMD science curricula, elementary teachers often remarked that 
although how they taught science had not changed, their attitudes about science were more 
positive and their enjoyment and excitement in teaching the subject had increased, reflecting a 
common research finding that teachers believe that they have “always done” what is required 
(Spillane & Zeuli, 1999). Middle and high school science teachers, on the other hand, saw 
changes in their pedagogy. As several teachers reported in focus groups: 

Prior to implementing [the product], his classes were two-thirds lecture and one- 
third activity based. During the activity-based portion, he might have an entire 
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class devoted to computers or wet lab or a hands-on activity. Now his classes 
are more fluid, and a single class will incorporate two or more of these aspects at 
a time. (Summarized from Teacher Interview, Project 16). 

The materials forced a less passive approach to teaching (that is, less of the “lec- 
ture, read, answer lab questions” format). Teachers could not use “cruise control” 
to teach. The exploratory labs and student questions dramatically changed the 
teaching of one teacher because the program requires her to answer student ques- 
tions when sometimes she doesn’t know the answer. (Summarized from Focus 
Group, Project 12). 

The high school course is very student oriented rather than being teacher oriented. 

There is far more emphasis on students’ sharing ideas in small groups and not on 
teacher lecture. The materials give guidelines for running the classes in this way. 

Changing the way of teaching was hard. As one teacher reported: “I had a rough 
time turning the room over to the kids. I have 23 years of being in total control 
with a lot of straight lecture. When I started [the course) that first year, turning the 
classroom over to the kids as a town meeting and saying, ‘Ok you have two days 
to plot out your diet and figure out food values’ — that was tough to do and 
maintain control.” ( Summarized from Focus Group, Project 2). 

Some science teachers who did not change their pedagogy experienced changes in their 
attitudes toward science based on their students’ and their own enjoyment of the materials: 

One teacher commented that [the product) has had an impact on her teaching in the 
sense that she had always found plant biology to be boring and really didn’t like 
teaching it, but now she looks forward to teaching the unit because her students like 
it so much and because they get so much out of it. (Summarized from Focus 
Group, Project 20). 

Comprehensive materials seem to be more likely than supplemental materials to have an 
impact on pedagogy, at least as the teachers reported it. The most common pedagogical changes 
noticed by teachers interviewed about IMD supplemental materials, was incorporating media, such 
as the Internet, videos, and special tools, into teaching. However, the teachers who used supple- 
mental materials identified and selected them and were likely to characterize themselves as using 
reform-oriented pedagogy prior to implementing the IMD materials. 

The new strategies make teaching more challenging for the teachers. The curriculum is no 
longer something they can simply pick up and implement, almost by rote. Instead, they need to 
study the materials in order to understand the concepts and strategies. Many teachers reported 
that far more preparation time was needed than was indicated by the publisher, including time to 
incorporate more group work into the day. 

Reform-oriented Pedagogy: Classroom Observations 

In addition to using focus group and interview data to collect teachers’ views on the impact 
IMD materials made on their pedagogical practice, WestEd and Abt Associates Inc. staff ob- 
served in classrooms in which IMD-funded comprehensive materials were used, guided by a 
structured observation instrument. The instrument addressed the overall quality of instruction, 
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mathematics or science content, classroom culture, and teacher behaviors. We found that both math- 
ematics and science classes were interactive environments in which hands-on activities were notable, 
although there was more variety in the science classes. 

Students in most mathematics classes we observed were engaged in problem solving and hands- 
on research. In elementary classrooms, we saw many student oral presentations. In contrast, there 
were few formal presentations by teachers. In fact, only at the middle school level did a majority of 
classes observed (70 percent) rely on formal teacher presentations. 

Across all criteria included in the instrument, the overall quality of the mathematics lessons 
observed was high, especially in elementary school. The instrument rates scores of 4 as “accom- 
plished, effective instruction” and 5 as “exemplary instruction,” and the average ratings were 4.5 
(elementary), 3.7 (middle school), and 2.7 (high school). The low high school rating is due to 
one class that was rated as exemplifying “ineffective” teaching. 

Observers also rated the content of the mathematics lessons highly, with average ratings of 
3.8 (elementary), 4.1 (middle school), and 4.2 (high school) on a five-point scale. The instru- 
ment addressed issues such as whether the content was appropriate, significant and worthwhile; 
accurate and relevant to the needs/interests of most students; and portrayed as a dynamic body of 
knowledge. 

The observers gave high ratings to items related to the culture of the classroom, especially 
the extent to which teachers encouraged active participation and collaboration among students 
and showed respect for and collaborated with students. Average ratings ranged from 3.7 to 3.9 
across the three school levels. 

Teacher behaviors, such as the teacher’s management style, pacing of the lesson, confi- 
dence in using the materials, and questioning strategies, were less highly rated. Averages ranged 
from 3.2 (high school) to 3.4 (elementary school) to 3.8 (middle school). 

One observer described an exemplary lesson in numeration and place values among first 
graders: 

Students are organized in stations, with the teacher leading one station. Students 
work on a problem and share solutions with the group. The problems cover the 
different skills of the unit and allow the teacher to assess if there are students who 
are having difficulty with the material. Students are engaged in a variety of tasks 
that make them responsible for their own learning and the learning of their peers. 

Most students are eagerly participating in the activities and in sharing their ideas 
with each other. There are enough manipulatives so that all students have the 
opportunity to work with them. When students arrive at different answers, they 
discuss their work and come to an agreement. The activities are well structured to 
provide students with the opportunity to practice what they have been learning in 
the unit. It also allows students to teach each other if there are concepts that are 
still unclear to some. 

Among the science classes observed, the major activities were hands-on science with some 
problem solving by students. Teachers made formal presentations in half or fewer of the classes. 
The elementary grade science classrooms received consistently high marks — 4.0 in overall 
quality, 3.8 each in content and in classroom culture, and 3.7 in teacher behaviors. Ratings for 
the middle and high school science classes were lower, with most average ratings at or near 3, 
the mid-point of the five-point scale. There was far more variability in ratings in science than in 
mathematics, with scores ranging from 1 (“activity for activity’s sake”) to 4 (“accomplished. 
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effective instruction”). 

In a highly rated elementary science class, where first graders were focusing on the properties of 
balls of clay, one observer wrote: 

The teacher was completely confident with the subject lesson. She kept the 
students engaged, asked them questions and really tried to have them come up 
with answers, encouraged them to think, and to try out other questions. The 
teacher continually came around as kids were working in groups to ask them 
questions and listen to what they were finding. Kids were asked to draw and 
write about what they found (both individually and in groups). The teacher then 
asked what was found, wrote findings on the board and led a discussion compar- 
ing the properties of balls. ...It was an example of what can happen with a great 
curriculum and a wonderful teacher. 

Among the lower scoring high school science classes, observers found that one teacher 
who had reportedly not bought into the IMD curriculum taught very traditionally, with emphasis 
on drill and practice. In another class, the observers wrote that the teaching was very “book- 
bound,” with the teacher neither giving students sufficient time to develop their own ideas nor 
pushing them academically. 

Student Engagement 

Although we have no independent measures of student engagement, teachers interviewed 
consistently spoke of increases in student engagement using the IMD materials, across compre- 
hensive and supplemental materials, grade levels, mathematics and science. 

Teachers of comprehensive mathematics materials said: 

Students are no longer afraid of math. “Students actually look forward to doing 
math” one teacher remarked. Another teacher remarked that she had a student 
who would cry whenever they did math, yet since using [an elementary school 
math program], the student is very happy. (Focus Group, Project 10). 

There is a better level of student engagement. Students have much more conver- 
sation with peers about math. As the year goes on, students tackle more compli- 
cated problems with less teacher help. The questions students ask are thoughtful 
rather than procedural. There is greater diversity in responses to questions. 

Students are more willing to take risks. (Focus Group, Project 19). 

“I like the way it engages students in mathematics. Students learn to be in control 
of their own learning. It’s not clear whether students are developing better math 
skills. What is clear is that students’ belief in their ability to do math is increasing 
dramatically. In their writing, they seem to think they understand math. For the 
first time, all students are moving on in math and enrolling in courses beyond 
what is required. This is remarkable, particularly for those students who have 
always struggled with and disliked math.” (Focus Group, Project 5). 

Teachers reported increased student engagement with supplemental materials for math- 
ematics as well. Videos provide a different angle for students that they can grasp right away and they 
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were reported to be fun to watch. One tool was highly praised because it allowed students to 
explore spatial relations and geometric properties that would have been very tedious by hand. 
Teachers reported that students see the results immediately, which helps attract and hold their 
attention. 

Science teachers also reported increases in student engagement: 

“The students do like using the materials. ...The subject holds their attention. 

They are solving problems and coming up with solutions, and they are having real 
discussions. The students come up with more questions, and even if they do not 
fully grasp the concept right away, the concept is on the way to being realized.” 

( Focus Group, Project 21). 

“Learning is coming from within, not from without. The lessons make science 
real for kids. It builds a strong conceptual understanding of basic science 
principles. The lessons are fun and really hold kids’ attention.” ( Focus Group, 
Project 22). 

“Subjectively, I think we’re doing a better job getting students more interested in 
coming to school on a daily basis. I get testimonials from parents to this effect.” 

( Focus Group, Project 4). 

“The students’ reactions are very positive. They do not want to be absent when 
the program is in use and they appear to be eager to learn as much as they can 
about how the cars are working. The students apparently go home and talk 
endlessly with their families about what they are learning.” (Teacher Interview, 

Project 20). 

Students become very attached to their [materials] and are extremely engaged in 
following the developmental process.... Two students with whom I was sitting had 
both named their materials, and according to them, almost all of their classmates 
had also done so. Both were very positive about the unit. One commented: “This 
is the best part of science, you know, where you get to do stuff in the labs.” The 
other student agreed, saying: “It’s better than if you watch a time lapse video 
because you’re doing it yourself and it’s yours and you can see everything happen- 
ing.” (Summarized from Classroom Observation and Focus Group, Project 21). 

Across both mathematics and science materials, teachers often reported that IMD materials 
were suitable for a broad range of students. Several special education teachers spoke of the 
usefulness of hands-on materials for their students. In addition, two middle school programs that 
integrated technology were given high marks for increasing the representation of girls in technology. 
In one site, two years ago, the seventh grade technology education class had only two girls, but 
with the IMD-funded materials, girls are half the students and are full participants. One anecdote 
illustrates a girl’s involvement in technology both within and outside the classroom: 

A student asked her father if she could have new bedroom furniture and her father 
replied that he didn’t think that she was mature enough. [We’re not sure why he 
thought that, but that’s the way the story is told.] While he was away on a business 
trip, she took the dimensions of her room and designed and developed a scale 
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model of the room, complete with what she had in mind — bed, dressers, study/ 
computer area, etc. — and presented it to her father when he returned home. Her 
father, impressed with her initiative and the detailed manner in which she had 
thought through and presented the design, agreed to buy her new furniture. The 
furniture arrived while her father was away on another business trip and her 
mother told her: “Well, I guess we’ll have to wait until your father gets back from 
his business trip to set up your new room.” The daughter was not so easily 
daunted as the mother and headed down to the basement to get all the tools she 
needed and started assembling the room on her own and had everything in place 
before her father returned home. Her teacher believes that this anecdote demon- 
strates some of the types of problem-solving skills that students use in the. . . 
classrooms. Furthermore, she described the student as a “princess” who, prior to 
taking the technology portion of the course, would never have dreamed of picking 
up hammers, wrenches, and screwdrivers to assemble her new room. (Summa- 
rized from Focus Group, Project 17). 

Student Academic Performance. 

Most of our information about student achievement comes from teachers’ assessment of 
student progress, although some teachers reported results from norm- or criterion-referenced tests 
in mathematics, with some limited information about possible impact coming from the classroom 
observations. Formal achievement data comes primarily from sites in which the materials were 
adopted at the district or school levels. 

Perhaps most strikingly, teachers said they assessed students differently from their previous 
practice. For example, teachers of an elementary-level mathematics curriculum noted: 

“[Before implementing this program] we probably assumed that students knew 
more than what they really did about certain concepts. Even though students 
(especially the high level students) were doing the problems and performing well, 
they often did not have a real understanding of what it was we were teaching. The 
explanation part of the materials made us take a closer look at the strengths and 
weaknesses of students.” (Focus Group, Project 10). 

Teachers gave multiple examples of their assessments of student learning, often focused on 
students explaining the strategies they have employed in solving problems. Several teachers also 
observed that students were applying strategies they learned to situations outside class (Project 
15 ). 

Teachers using the IMD science materials also reported that students were more confident in 
themselves as learners, were now more independent and self-starting, and had built their social 
skills because of the need to gather information and make presentations as part of small groups. 

A middle school teacher echoed comments from other middle and high school teachers about what 
students learn from IMD science materials: 

These materials encourage students to develop and test their own 

hypotheses He finds that his students are now more willing to take these kinds 

of risks. He finds that girls, in particular, seem to talk more in class. He used the 
example of a girl in his class who collected data and believed that she had refuted 
a widely accepted scientific theory. She was confused and thought she had made 
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a mistake, but wasn’t sure where or how. She came to the teacher to talk 
through what she had done. As she explained what she had done, she figured out 
what she had done wrong. According to the teacher: “She thanked me for 
listening, but she solved the problem herself. Students get into the process 
enough to develop a gut instinct and say to themselves ‘Okay, I need to redo this 
and this and this.’ Stuff like that is good for kids. They’re not so dependent on 
us for what they understand.” (Summarized from Focus Group, Project 16). 

Although teachers of the IMD-funded mathematics materials reported higher levels of 
sophistication in thinking mathematically, some worried that students’ computational skills were 
inadequately addressed at several grade levels. 

Several implementing sites assessed student achievement on the comprehensive IMD- 
funded mathematics products through state or district norm-referenced tests. These are cross- 
sectional designs (e.g., 8th grade students in one year compared with 8th grade students in the 
next) with no comparison group. Hence, attributing changes in achievement to the products and 
not to other factors is problematic. 

For the two comprehensive elementary school mathematics programs for which users 
reported data (Projects 10 and 15), test scores have improved. One experienced an initial drop 
in standardized test scores and then gains. Teachers reported being a little surprised by the gains 
because they were on norm-referenced tests, and they decided to readopt the materials on the 
basis of the gains. 

At the middle school level, all three comprehensive math programs had achievement data. 
The district adopting Project 19 looked at cross-sectional gains in sixth graders and eighth 
graders over several time points, and this year the district is beginning a longitudinal study to 
track the same students for five years. Increases in problem-solving skills among upper middle 
school students was attributed to Project 11, although some in the district think that increased 
mathematics scores could be related to the district’s paying more attention to mathematics in 
general. In another district, advocates attributed increases in student achievement on the state 
assessment to the middle school math program (Project 1), although other teachers claimed that 
differences in the cohorts of middle school students could have been responsible. At the high 
school level, Project 5 showed improvements in test scores but as with other levels, attribution is 
difficult. 

Once products were adopted by a district or school, cross-product comparisons of a 
product’s relative impact on student learning were rarely made. Four of the seven comprehen- 
sive IMD math products were adopted either district- or school-wide, so no other products 
were available as comparisons. A fifth product was adopted as an alternative to the standard 
curriculum, but no achievement comparisons have been made. Another comprehensive high 
school math curriculum was also adopted as an alternative to the standard curriculum, with 
about 20 percent of students enrolled. The district visited is just beginning to assess the relative 
effectiveness of the two curricula through student grades, persistence in taking math courses, 
SAT/ACT results, and state test results. The final district in this group was cited in the case 
study of adoption, above. 

Among the nine comprehensive IMD science products, none conducted formal student 
achievement assessments, nor was student achievement compared with other products. No 
student performance data were available about supplemental materials, either in math or science. 
This is not surprising because such assessments would have required considerable methodologi- 
cal sophistication. 
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Classroom observers were asked to rate the “likely effect” of the lesson observed on 
students’ understanding and self-confidence. We found that observations supported the finding 
that better implemented materials were deemed to be likely to affect student outcomes more 
than poorly implemented materials. 

Elementary mathematics classes received high ratings in such areas as student understanding of: 
mathematics using multiple approaches (4.3 on a five-point scale); science as a dynamic body enriched 
by investigation (4.0); and the importance of mathematics and science concepts (3.5). Observers also 
gave high marks to the likely effects of the lesson on student self-confidence in doing mathematics (4.2). 
At the elementary school level, the IMD products received higher ratings than did the non-IMD prod- 
ucts. 

The ratings of middle school mathematics classes were also very high (3.9 to 4.4) except in 
mathematics as a dynamic body of knowledge (3.2), but high school mathematics ratings were 
generally around 3, the mid-point on the five-point scale, and lower than elementary or middle 
school. However, the high school IMD mathematics ratings were higher than those given in 
classrooms using non-IMD materials, particularly with regard to likely effects on student under- 
standing of important mathematics concepts (3.7 vs. 2.9); of mathematics as a dynamic body of 
knowledge enriched through investigation (3.0 vs. 1.5); and on students’ self-confidence in doing 
mathematics (3 vs. 2.5). 

The likely effects of the IMD science lessons were all above 3.5 at the elementary school 
level, around 3 at the middle school level, and between 2.5 and 3.2 at the high school level. 

These lower scores parallel the generally lower implementation ratings that science curricula 
received from the classroom observations. 

Non-IMD Products 

The information presented in this section must be interpreted with care. Few teachers of 
the non-IMD materials were as conscious of their own pedagogy as were many of the IMD 
teachers, and fewer of them reported changes in practice. We were intrigued by the difference in 
self-reflection between teachers of IMD products and those using non-IMD products, but the 
finding must be placed in the context of the fact that the teachers we observed and interviewed 
were “nominated” by developers or publishers. They may have been selected because of their 
reflective natures, reflective teachers may gravitate to reform, or the materials may stimulate 
reflection. Additional research is necessary to resolve the matter. 

Further, although the observations illustrate that teachers of non-IMD materials tended to 
use more traditional instructional techniques, the difference may not be attributable to the prod- 
ucts in use. Indeed, at least one teacher was observed using an IMD-funded and a non-IMD 
product, and her practice was the same with both. In short, far fewer teachers using non-IMD- 
funded curricula than IMD-funded curricula reported impacts on: 

• Content knowledge 

• Reform-oriented pedagogy 

• Student engagement 

• Student outcomes 
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Each of these issues will be discussed in the following section. 

Content Knowledge 

Only one teacher of non-IMD-funded materials reported any impact of the materials on content 
knowledge: 

The curriculum has made it much easier to teach science. Science is not my 
strength, so it was more difficult for me to get children excited about learning 
science. [The curriculum] has become the basis of how I teach. 

This teacher’s experience echoes that of elementary science teachers using several of the 
IMD-funded elementary science curricula, described above. 

Reform-oriented Pedagogy 

Far fewer teachers using non-IMD-funded curricula reported that the materials had an 
impact on their instructional approaches, which is not surprising because the materials are quite 
traditional, or only moderately reform-oriented. One teacher using one of the more reform- 
oriented curricula noted the following change in her instructional approach: 

The roles reversed. Before we would stand up in front and say “Okay, this is the 
program. This is the answer. This is how I got it — now you do the same thing.” 

Now it’s the reverse. “You have to come up with the answer and you have to 
explain it to me.” ( Teacher Interview, elementary school non-IMD mathematics 

curriculum.) 

Another teacher using an integrated mathematics curriculum reported incorporating more 
activities and making better connections between mathematics and science than in her previous 
practice. 

Other teachers believed the materials they were using supported appropriate classroom 
practice, but the real motivation for their pedagogy came from other factors. One chemistry 
teacher noted, for instance, that most of his students are not there for the fun of chemistry, so he 
has had to do everything that he can to draw them in and engage their interest. Although the 
variety of approaches in the curriculum helps, he believes it is really up to him to engage their 
interest. Teachers in another district saw new trends in education and the influence of a “change 
agent” teacher in their school as the forces behind their move to more concept-driven and hands- 
on approaches to learning rather than the materials they are using. 

Classroom observations revealed two differences in the use of IMD and non-IMD materi- 
als. First, the classrooms using non-IMD materials were more likely to have formal teacher 
presentations than were those using IMD-funded materials. Observers indicated that 75 percent 
of all non-IMD classes included formal presentations as a major activity, although about half also 
had students engaged in hands-on research and problem-solving work. Second, non-IMD classes 
were less likely to use technology. Although few observers in either IMD or non-IMD classes 
rated technology use as a “major” activity, the non-IMD classes had less use than did the IMD 
classes. 
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Student Engagement 

Teachers using non-IMD materials provided us with fairly few examples of impacts on 
student engagement, in contrast to the many testimonials offered by teachers using IMD-funded 
materials. The examples all come from teachers in schools implementing more reform-oriented 
curricula. 

They enjoy the math class more when we do it, and I will forever use it. ( Teacher 
Interview, reform-oriented non-IMD mathematics curriculum). 

[The curriculum] does an excellent job in making students feel good about them- 
selves and their ability to do math. (Teacher Interview, reform-oriented non-IMD 
mathematics curriculum). 

One teacher, however, questioned whether increasing student enjoyment was an appropriate goal: 

With integrated mathematics, it’s a way of making the kids enjoy it more, but I am 
not convinced that it helps them any more. I would like to see evidence that inte- 
grated works better than the traditional methods. Publishers need site studies 
showing their curriculum’s effectiveness. (Teacher Interview, non-IMD high 
school mathematics curriculum). 

Except in one school, where teachers reported students as “hating” their middle school 
science text, teachers using more traditional curricula gave fairly curt responses, indicated the 
curricula meet their students’ needs, and students either liked the materials or at least thought that 
they were “okay.” 

Academic Performance 

None of the districts we visited shared information about student achievement on either 
norm- or criterion-referenced tests. Two districts using integrated mathematics curricula re- 
ported that more students in their districts are taking mathematics. One district, in fact, had 
adopted the integrated mathematics because they wanted to increase the numbers of students 
taking advanced mathematics. Although pleased with increased enrollment, they will not con- 
sider the adoption a success until they have some evidence that test scores are also improving. 

Conclusion 

IMD-funded products are designed to change how teachers think about science and math- 
ematics, as well as how the content and pedagogy they employ when working with students. 
Comprehensive materials are more likely to have an impact on teacher conceptions and practice 
than are supplementary materials. 

When teachers do not implement the materials as designed, the impact on classroom 
practice and student engagement is decreased. In contrast, however, when the IMD materials 
surmount the many barriers to appropriate implementation, classrooms are places in which 
students are highly engaged in important learning activities. We saw a number of such examples 
in the study. Consequently, we believe that the gap between project intention and actual use and 
impact comes less from the materials themselves than from the contexts in which they are used. 
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The strength of the products is shown by the fact that almost all the IMD materials were well imple- 
mented in a few places, and all had positive results when they were. 
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CHAPTER VIII 



Conclusions and Recommendations 

The evaluation of the IMD program focused on questions related to the development, market- 
ing, adoption, implementation, and impact of the materials supported by the program. The lenses 
through which we looked put the materials at the center of reform of mathematics, science, and 
technology education. As such, the study may have slighted alternative perspectives and the role of 
other NSF programs in reform. Indeed, we found that many sites with the strongest implementations 
were found in states that had participated in Statewide Systemic Initiative efforts. 

Our findings, in brief, were that the materials were of high quality, but that marketing, adop- 
tion, and implementation were problematic. As a result, there was deviation between intention and 
actuality at every link in the chain between development and impact. Products were marketed by 
small publishers who had limited resources for aggressive dissemination, so they were unable to 
increase the market for reform-oriented materials through their efforts. Further, adoption process 
frequently involved single teachers so widespread use in a district was rare. Teachers became 
aware of the materials through their participation in pilot or field tests or by seeing them at subject- 
matter conferences. Although such teachers were able to use school and district funds to purchase 
the materials, in general, they were unable to influence other teachers to use them. Also, imple- 
mentation encountered problems due to lack of sustained professional development; and impact 
was more limited than it could be. On the other hand, most products overcame the hurdles at each 
juncture and were successfully implemented in at least a few settings. The lessons from these 
successes and failures can guide future NSF programming. 

This concluding chapter begins by summarizing the answers to the questions posed by the 
study, and then moves to recommendations for NSF action. The recommendations place the 
IMD program within a portfolio of activities NSF can support, whether directly through materi- 
als development or through other mechanisms. 

Answers to the Evaluation Questions 

The study was framed by six questions: 

1 . To what extent do instructional materials embody the national standards, 
including an emphasis on thinking skills and making connections across 
curriculum topics? 

2. To what extent do they reflect what is currently known about good instruc- 
tional practice? 

3. How well have they been marketed? 

4. To what extent do adopters and teachers use the materials? 

5. What supports do teachers and other school-based professionals need to 
make the best use of the materials? 

6. What is the impact of the materials on classroom practice? 
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The first two questions address issues over which NSF exerts direct influence by creating 
appropriate guidelines for funding, including reviewers who understand the intent of the program, 
and funding projects that hold promise of embracing the standards and reform-oriented peda- 
gogy. The other questions are about the downstream issues, and the IMD program itself is more 
limited in the actions it can take about these matters, at least at the time of funding. In addition, 
the findings of the study related to these questions are more suggestive than definitive. 

Each question will be answered in turn. 

The Quality of the Materials 

Both members of the Expert Panel and teachers who use the materials rate them highly. 

They believe that the materials embody the national standards. However, the quality of the materi- 
als is not sufficient to lead to widespread use. First, many teachers and parents do not embrace 
the reforms inherent in the products, and even in adopting districts, may resist their use or change 
their nature during implementation. For example, we conducted focus groups with teachers who 
“reorganized” the curriculum to make it more like what they were used to, ignoring the developers’ 
intent. Second, state and local standards and tests are not always congruent with the national 
mathematics and science standards, and teacher success is judged by how well their students 
perform according to state and local expectations. Nonetheless, the IMD program goal of creating 
high quality materials has been realized. 

Instructional Practice 

IMD-supported materials are highly rated as supporting what is currently known about good 
instructional practice. Both the Expert Panel and teachers who are using the materials agree that 
the materials embrace reform-oriented pedagogy. Teachers reported that they encouraged student 
inquiry, problem solving, and making connections across curriculum topics as they used the materi- 
als. 

However, despite the positive ratings related to instructional practice, the materials faced 
barriers in implementation. For example, users of IMD products reported that parents were 
concerned that student homework (if it existed) was not what they expected. Some teachers then 
“supplemented” the materials with drill and practice homework sheets. In other instances, com- 
prehensive materials were used as supplementary, and reform-oriented pedagogy became 
marginalized in the classrooms. 

However, the materials themselves are appropriate models that reflect current thinking 
about instruction. 

Marketing 

The first stumbling blocks to program success appear at the link between developers and 
marketers. When the relationship is positive, the two share a vision and work closely to market 
the materials, but both developers and publishers frequently cited misunderstandings or differences 
in perspectives. Further, both report being hampered by what they believe is a limited market for 
reform-oriented materials. As a result, major publishers are unlikely to publish the materials, and 
the smaller, reform-oriented publishers have fewer resources to use for marketing. 

The marketing problem arises because of the limited market for the materials, but is exacer- 
bated by the fact that the most effective marketing tool is professional development. Professional 
development helps sell products in two different ways. First, developers, marketers, and adopters 
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agree that seeing the materials in a workshop setting is more likely to stimulate sales than any 
other approach. One person drew the analogy of the difference between reading a menu and 
eating a meal and other methods of dissemination and actual experience with the product. 
Second, and equally important, adopters cited the availability of professional development as an 
important factor in their decision to adopt. In the “ideal case” cited above, the key advocate 
said, “Service sold it,” referencing both workshops and ongoing support to teachers. 

Professional development is expensive, and publishers believe that their role is to provide 
support at the front end of use. They argue that others are responsible for ensuring that teachers 
have adequate knowledge of the content and are able to use a variety of instructional ap- 
proaches. If teachers came to classrooms with strong preparation in content and pedagogy from 
their preservice education, there might be less need for the depth of professional development 
required by the materials. The professional development issue is exacerbated by the fact that 
most IMD publishers are small and do not have the resources for significant amounts of front- 
end support, nor should they be placed in a position that requires them to do more than help 
teachers use particular materials. 

We found that marketing is most successful when it reaches those who are already interested 
in reform and acquainted with current instructional approaches. For this reason, teachers who 
were involved in pilot and field tests frequently became a major source of sales of the materials. 

Use of the Materials 

There was considerable slippage between marketing, adoption, and use of the materials. 
When the process worked well, teachers and community members were invested in the success 
of the product and used it as well as they could. Adoption processes that engendered such 
investment varied, depending on the setting. For example, some districts built support through 
adoption committees, but there were fewer of these than anticipated. In other sites, an individual 
teacher began using the product and her/his enthusiasm (and the enthusiasm of students) spread 
its use. Whatever the process, if it yielded understanding of and support for the approaches taken 
by the materials, successful use followed. 

We found that barriers to implementation were the mirror image of the facilitators. For 
example, technology could support use if it were available but was a barrier if the product relied 
too heavily on the existence of large numbers of computers. However, if materials that use 
technology are not available, there will be little reason for schools and districts to invest in 
computers. 

Perhaps the most important barrier to use lay in the fact that teachers sometimes resisted 
use or used the materials in ways that conflicted with the developers’ intention when they lacked 
the skills and knowledge related to content and instructional practice. Such resistance and 
misuse arose less in sites in which there was strong support for appropriate implementation. 

Required Support 

As indicated throughout this report, sustained professional development is a necessary 
accompaniment to successful implementation of IMD products. The materials themselves 
require teachers to change their conceptions of mathematics, science, and technology as aca- 
demic content areas, moving from an emphasis on passive knowledge of facts and algorithms to 
active construction of knowledge. In addition, the materials embody instructional approaches that focus 
on the student as problem solver and thinker. These changes place great demands on teachers, and 
professional development is an essential tool for their learning how best to use the materials. 
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The sustained professional development associated with successful implementation had a number 
of components. First, teachers were introduced to the materials through some kind of hands-on work- 
shop. Second, extended institutes or workshops, most often held during the summer, gave teachers an 
opportunity to practice using the materials in instruction. Third, someone — either a developer, publisher 
representative, or teacher who was more experienced in the program — was available to provide 
support during the school year. And, finally, in the best circumstances, the developer received feedback 
and changed or supplemented the materials to meet teachers’ needs. 

In addition to formal professional development, successful implementers often had an advocate or 
champion. The champion was able to arrange events with parents so they were supportive of the 
curriculum change. Further, he/she solved logistical problems by such actions as developing a central- 
ized location for laboratory materials or manipulatives and a system for their use that facilitated teacher 
access to the materials. Having a champion meant that the teachers using the materials believed they 
would get help, which increased their enthusiasm for the products. 

Impact 

When materials were well implemented, they had positive impacts on classrooms, as 
reported by teachers. For example, elementary school teachers reported increased knowledge of 
science, and, to a lesser extent, of mathematics as a result of using the materials. Elementary, 
middle, and high school teachers said that the materials helped them use more student-centered, 
problem solving instructional approaches. And teachers at every level reported increased student 
enthusiasm for science, mathematics, and technology. 

Only sites implementing mathematics curricula had data related to student achievement, 
typically from state or district tests. The designs did not include comparison groups so attribu- 
tion is problematic. However, those that had information about student learning indicated posi- 
tive outcomes. 



Recommendations 

The recommendations arising from the evaluation of the IMD program fall into two broad 
categories. The first focuses specifically on actions IMD program staff can take that will improve 
product development and dissemination. The second group of recommendations concerns issues 
outside the direct influence of the IMD program, and focuses on creating the climate and support 
for reform that we found essential to successful implementation and positive impact. 

IMD Program Recommendations 



IMD program guidelines should emphasize the importance of including 
teachers on development teams. 

IMD program staff should acknowledge the importance of individual 
teachers as adopters of IMD products and seek ways to increase their oppor- 
tunities to influence other teachers, through support for networks and school- 
and district-reform efforts that build on teacher knowledge, skills, and interest. 

IMD program guidelines should increase the stress on evaluation as a compo- 
nent of the development process, and should also emphasize the importance of 
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collecting data related to student learning of mathematics, science, and 
technology on tests not directly connected with the materials, but appropri- 
ate to them, such as the NAEP or TIMMS. 

• IMD program staff should offer assistance to developers through such 
avenues sessions at Principal Investigator meetings and other venues as they 
begin to work with publishers and develop relationships with agents who 
work with publishers. 

• IMD program staff and grant recipients should strengthen connections with 
other NSF and federal programs to increase opportunities to connect with 
professional networks and create awareness of the products. Such avenues 
as the NSF Implementation Centers and the various systemic change 
projects, the Eisenhower Regional Consortia, and the Eisenhower National 
Clearinghouse are potentially useful to dissemination. 

• NSF should sponsor a study that proceeds from an alternative vision for the 
IMD program — that the purpose of the program is to demonstrate what 
types of materials are possible and influence professional practice, 
preservice education, and publishers through the example the materials 
present. 

Supporting Reform More Broadly 

• NSF should stimulate a public dialogue about reform of mathematics, 
science, and technology education and the role of materials in efforts to 
improve education. 

• NSF should provide support to professional networks, including the Presi- 
dential awardees, in order to increase their effectiveness in stimulating 
attention to reform of mathematics, science, and technology education. 

• NSF should work with professional associations outside the science and 
mathematics community to build support for reform. Such organizations as 
the American Association of School Administrators (AASA), the Associa- 
tion for Supervision and Curriculum Development (ASCD), and the Ameri- 
can School Boards Association (ASBA) offer opportunities to work with a 
broad array of educators and policymakers to create the environment that 
will support the use of the materials. 

• NSF should increase attention to support structures and approaches that 
provide sustained professional development related to reform-oriented 
curricula and pedagogy. Such professional development should include in- 
depth workshops and institutes and on-site support to teachers. 

• NSF should increase attention to the role of materials in preservice educa- 
tion. Teachers play a major role in choosing instructional materials, and 
most current preservice courses do not include work on processes and 
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criteria to use in making such choices. In addition, preservice teachers 
who are familiar with reform-oriented materials are likely to be supportive 
of their use when they begin their teaching careers. 
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APPENDIX 1 



Expert Panel 

Sam Alessi (public school administrator, Buffalo, New York) 

Sigmund Abeles (retired public school administrator, Connecticut) 

Sarah Lee Armstrong (college biology, Millsaps College) 

Ramesh Gangolli (university mathematics, University of Washington) 

Barbara Janson (former publisher and independent consultant, Massachusetts) 
Martin Johnson (mathematics educator, University of Maryland) 

Charlotte Keith (middle school mathematics teacher, Olathe, Kansas) 

Ramon Lopez (university science, University of Maryland) 

Barry Rowe (public school technology, Champaign, Illinois) 

Ernest Savage (university science, Bowling Green State University) 

Ethel Schultz (retired science educator, Massachusetts) 
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e. What are the major domains/topics of content covered by these materials? 
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APPENDIX 2 



Instrument For Evaluating Materials 



Framework for Review 

Instructional Materials for Elementary School Science 1 



Title: 

Author(s): 

Publisher: Copyright date: 

Reviewed by: Date: 

I. Descriptors 

a. Write a brief description of the components of the curriculum upon which this review is based 
(e.g., teachers guide, student books, hands-on materials, multimedia material). That is, what 
materials did you receive and include in your review? 



b. Write a brief description of the purpose and broad goals of these materials. That is, what were 
the stated purposes and what were the actual purposes of the materials? 



c. What grade levels do the materials serve? 

K 1 2 3 4 5 



d. Are the instructional materials designed to 

provide a complete multi-year program for elementary school science. 

provide a complete one-year course for elementary school science. 

provide multiple modules or units that could be used to supplement other course materials 

for elementary school science. 

provide a single module or collection of activities that could be used to supplement other 

course materials for elementary school science. 

other (explain): 



1 NOTE: This framework is adapted from an instrument developed by Inverness Research under contract to the 
National Science Foundation. The framework was refined as part of a panel review of NSF -supported materials for 
middle school science, which was limited to projects that provide at least a year-long course of study. 
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II. Quality of the Science 



Directions: For each item, circle the number corresponding with your response to the question. 
Write an explanation for your rating of each item below the item. 



a. Does the content in the instructional materials align well with all eight areas of the Content 
Standards as described in the National Science Education Standards (NSES)? 

(See attached guidelines) 



1 2 3 

Omits substantial content Some misalignment 

included in NSES of content with 

and/or includes substantial content recommendations 

not recommended in NSES in NSES 



The curriculum 
aligns well with 
content recommendations 
in NSES 



b. Are the science concepts presented in the instructional materials accurate and correct? 
[Provide examples of major errors where they are evident. Attach extra page if necessary.] 

1 2 3 4 5 

Substantial, major errors Mostly correct, with Scientifically accurate, 

some minor errors and correct 



c. Do the instructional materials adequately present the major concepts in the standards and 
adequately demonstrate and model the processes of science? 

1 2 3 4 5 

Major concepts and Major concepts and Major concepts and 

processes not addressed processes somewhat addressed processes addressed well 



d. Does the science presented in the instructional materials reflect current disciplinary knowl- 
edge? 

1 2 3 4 5 
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The ideas are out of date 



Somewhat current 



Current 



e. Do the instructional materials accurately represent views of science as inquiry as described in 
the National Science Education Standards? 



1 

Poor examples 
of inquiry 


.2 3 4 5 

Mixed Rich and accurate 

quality examples of inquiry 



f. Do the instructional materials accurately present the history of science? 



i 

Poor portrayal 
of history of science 


.2 3 4 5 

Mixed Rich and accurate 

quality portrayal of history of 

science 



g. Do the materials emphasize technology as an area of study? 



i 

Little or no 
emphasis 


. .2 3 4 5 

Some emphasis Rich and well 

designed emphasis 



h. Do the materials emphasize the personal and societal dimensions of science? 



i 

Little or no 
emphasis 


. .2 3 4 5 

Some emphasis Rich and well 

designed emphasis 


o 


89 

Final Report on the Evaluation of the National Science Foundation’s Instructional Materials Development Program 



i. Do the materials emphasize the content of life science? 



1 2 

Little or no 
emphasis 



. .3 4 

Some emphasis 



5 

Rich and well 
designed emphasis 



j. Do the materials emphasize the content of earth science? 



1 


2 


3 


4 


5 


Little or no 




Some emphasis 




Rich and well 


emphasis 








designed emphasis 



k. Do the materials emphasize the content of physical science? 

1 2 3 4 5 

Little or no Some emphasis Rich and well 

emphasis designed emphasis 



1. Do the instructional materials provide sufficient activities for students to develop a good 
understanding of key science concepts? 



i 


. .2 


3 4 


5 


Too few learning 




Activities provide 


Activities provide 


activities 




some opportunity 


many rich opportunities 






for students to learn 


to learn key science 






some important concepts 


concepts 
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m. Do the instructional materials provide sufficient opportunities for students to apply their 
understanding of the concepts (i.e., designing of solutions to problems or issues)? 

1 2 3 4 5 

Very few Some Very rich in 

application activities application activities application activities 



n. Do the instructional materials present an accurate picture of the nature of science as a dy- 
namic endeavor? 



1 2 

The image of science is 
out-of-date, inaccurate, 
or non-existent. 



. .3 4 . 

The image of 
science is of 
mixed quality. 



5 

The image of 
science is current 
and accurate. 



o. Do the materials develop an appropriate breadth and depth of science content? 



1 2 . 

Too narrow 
or too broad 



. .3 

Somewhat 

balanced 



Good balance of 
breadth and depth 



p. What is the overall quality of the science presented in the instructional materials? 
1 2 3 4 5 

Low Medium High 
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III. The Pedagogical Design 



a. Do the instructional materials provide a logical progression for developing conceptual under- 
standing in science? 



1 

No logical 
progression 
of ideas 



2 



3 

Somewhat logical 
progression of ideas 



Logical progression 
of ideas that builds 
conceptual understanding 



b. Do the instructional materials provide students the opportunity to make conjectures, gather 
evidence, and develop arguments to support, reject, and revise their preconceptions and explana- 
tions for natural phenomena? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 



c. To what extent do the instructional materials engage students in doing science inquiry? 



1 2 , 

Very few or very 
contrived activities 
for students to do 
science inquiry 



3 

Some good activities 
for students to do 
science inquiry 



Many rich and authentic 
opportunities for 
students to do 
science inquiry 



d. To what extent do the instructional materials engage students in doing technology problem 
solving? 



1 2 

Very few or very 
contrived activities 
for students to do 
technology 
problem solving 



3 

Some good activities 
for students to do 
technology 
problem solving 



Many rich and authentic 

opportunities 

for students to do 

technology 

problem solving 
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e. To what extent does the curriculum engage students in activities that help them connect 
science to everyday issues and events? 



1 2 

Very few or very 
contrived activities 
for students 
to make connections 



3 

Some good activities 
for students to make 
connections 



.4 5 

Many rich and authentic 
opportunities 
for students 
to make connections 



f. How would you rate the overall developmental appropriateness of the instructional materials, 
given its intended audience of ALL students at the targeted level(s)? 

1 2 3 4 5 

Not developmental^ Somewhat developmental ly Developmentally 

appropriate appropriate appropriate 



g. Do the materials reflect current knowledge about effective teaching and learning practices 
(e.g., active learning, inquiry, community of learners) based on research related to science educa- 
tion? 

1 2 3 4 5 

Do not reflect Somewhat reflective Reflect well 

current knowledge of current knowledge current knowledge 

about teaching and learning about teaching and learning about teaching and 

learning 



h. Do the instructional materials provide students the opportunity to clarify, refine, and consoli- 
date their ideas, and to communicate them through multiple modes? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 
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i. Do the instructional materials provide students the opportunity to think and communicate 
scientifically? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 



j. Do the instructional materials provide students with activities connecting science with other 
subject areas? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 



k. Are the instructional materials likely to be interesting, engaging, and effective for students? 
1 2 3 4 5 

Not at all interesting Somewhat interesting Interesting and engaging 



1. Are the instructional materials likely to be interesting, engaging, and effective for girls and 
boys? 

1 2 3 4 5 

No sensitivity Some sensitivity Sensitive to 

to gender issues to gender issues gender issues 



m. Are the instructional materials likely to be interesting, engaging, and effective for 
underrepresented and underserved students (e.g., gender, ethnic, urban, rural, with disabilities)? 

1 2 3 4 5 

No sensitivity Some sensitivity Sensitive to 

to underrepresented and to underrepresented and underrepresented and 

underserved students underserved students underserved students 
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n. Does assessment have explicit purposes connected to decisions to be made by teachers (e.g., 
prior knowledge, conceptual understanding, grades)? 

1 2 3 4 5 

Unclear purposes Somewhat clear Clear statement 

purposes of purposes 



o. Do assessments focus on the curriculum’s important content and skills? 

1 2 3 4 5 

Poor correspondence Fair correspondence Full correspondence 



p. Do the instructional materials include multiple kinds of assessments (e.g., performance, 
paper/pencil, portfolios, student interviews, embedded, projects)? 

1 2 3 4 5 

Little or no student Some variety of Complete 

assessment provided student assessment student assessment 

package 



q. Are the assessment practices fair to all students? 

1 2 3 4 5 

Fair for a few Fair to most Fair to all 



Q 
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r. Do the instructional materials include adequate and appropriate uses of a variety of educa- 
tional technologies (e.g., video, computers, telecommunications)? 



1 2 

Little or no 

educational technology 
included 



3 4 , 

Some appropriate 
educational technology 
included 



5 

Many appropriate 
applications of 
educational technology 
included 



s. What is the overall quality of the pedagogical design of these instructional materials? 

1 2 3 4 5 

Low Medium High 



t. To what extent are the purposes of the materials clear to students? 

1 2 3 4 5 

Purposes are unclear Purposes are somewhat clear Purposes are clear 
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IV. Implementation and System Support 

a. Will the teachers find the materials interesting and engaging? 

1 2 3 4 5 

Dry and boring Somewhat interesting Interesting and engaging 

and engaging 



b. Do the instructional materials include information and guidance to assist the teacher in imple- 
menting the lessons? 



1 


...2 


3 


...A.... 


5 


No teacher support 




Some teacher support 




Rich and useful teacher 










support 



c. Do the instructional materials provide information about the kind of resources and support 
system required to facilitate the district implementation of the science materials? 

1 2 3 4 5 

No materials support Some materials support Rich and useful materials 

support 



d. Do the instructional materials provide information about how to establish a safe science 
learning environment? 

1 2 3 4 5 

No safety information Some safety information Rich and useful safety 

information 
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e. Do the instructional materials provide information about the kinds of professional develop- 
ment experiences needed by teachers to implement the materials? 

1 2 3 4 5 

Little or no Partial information Rich and useful 

information provided provided information 

provided 



f. Do the materials provide guidance in how to link the materials with the district and state assessment 
frameworks and programs? 

1 2 3 4 5 

No guidance Some guidance Rich and useful 

guidance 



g. Do the materials provide guidance and assistance for involving administrators, parents, and the 
community at large actively in supporting school science? 

1 2 3 4 5 

No guidance Some guidance Rich and useful 

guidance 



h. Overall, are the materials usable by, realistic in expectations of, and supportive of teachers? 

1 2 3 4 5 

Teacher unfriendly Somewhat teacher friendly Teacher friendly 
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V. Major Strengths and Weaknesses 

a. In your opinion what are the three major strengths of this curriculum? 



b. In your opinion, what are the three major weaknesses of this curriculum? 
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VI. Overall Quality, Value, and Contribution 

a. In your opinion what is the overall quality of these materials relative to: 

low high 



- turning students on to science? 

- making students think? 

- quality of science content? 

- quality of pedagogy? 

- quality of classroom assessments? 

- encouraging teachers to teach differently? 



1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



b. In your opinion, what is the overall quality of these instructional materials? 

1 2 3 4 5 

Low Medium High 



c. To what extent would you encourage the dissemination, adoption, and implementation of this 
curriculum? 



1 2 

Not worthy of 
dissemination, adoption, 
nor implementation 



3 

OK to disseminate, 
adopt, and implement 
if revised 



OK to disseminate, 
adopt, and implement 
as is 
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Guidelines for Section II. 



The following is a brief outline of the National Science Education Standards. It should be used 
to guide your responses to Section II. 



CONTENT STANDARD 


A. Science as Inauirv 


1. Abilities necessary to do science inauirv 


2. Understandings about scientific inauirv 


B. Physical Science 


1 . Properties of objects and materials 


2. Position and motion of objects 


3. Light, heat, electricity, and magnetism 


C. Life Science 


1 . The characteristics of organisms 


2. Life cycles of organisms 


3. Organisms and environments 


D. Earth and Space Science 


1 . Properties of Earth materials 


2. Objects in the skv 


E. Science and Technology 


1. Abilities to distinguish between natural obiects and objects made bv humans 


2. Abilities of technological design 


3. Understandings about science and technology 


F. Science in Personal and Social Perspectives 


1 . Personal health 


2. Characteristics and changes in populations 


3. Types of resources 


4. Changes in environments 


5. Science and technology in local challenges 


G. History and Nature of Science 


1 . Science as a human endeavor 


H. Unifying concepts and processes 


1 . Order and organization 


2. Evidence, models, and explanation 


3. Change, constancy, and measurement 


4. Evolution and equilibrium 


5. Form and function 
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Framework for Review 

Instructional Materials for Middle School Science 1 



Title: 

Author(s): 

Publisher: Copyright date: 

Reviewed by: Date: 



I. Descriptors 

a. Write a brief description of the components of the curriculum upon which this review is based 
(e.g., teachers guide, student books, hands-on materials, multimedia material). That is, what 
materials did you receive and include in your review? 



b. Write a brief description of the purpose and broad goals of these materials. That is, what were 
the stated purposes and what were the actual purposes of the materials? 



c. What grade levels do the materials serve? 

5 6 7 8 



d. Are the instructional materials designed to 

provide a complete multi-year program for middle school science. 

provide a complete one-year course for middle school science. 

provide multiple modules or units that could be used to supplement other course materials 

for middle school science. 

provide a single module or collection of activities that could be used to supplement other 

course materials for middle school science. 

other (explain): 



1 NOTE: This framework is adapted from an instrument developed by Inverness Research under contract to the 
National Science Foundation. The framework was refined as part of a panel review of NSF-supported materials for 
middle school science, which was limited to projects that provide at least a year-long course of study. 

Final Report on the Evaluation of the National Science Foundation’s Instructional Materials Development Program 

103 



e. What are the major domains/topics of content covered by these materials? 
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II. Quality of the Science 



Directions: For each item , circle the number corresponding with your response to the question . 

Write an explanation for your rating of each item below the item . 

a. Does the content in the instructional materials align well with all eight areas of the Content 
Standards as described in the National Science Education Standards (NSES)? 

(See attached guidelines) 



1 2 3 

Omits substantial content Some misalignment 

included in NSES of content with 

and/or includes substantial content recommendations 

not recommended in NSES in NSES 



The curriculum 
aligns well with 
content recommendations 
in NSES 



b. Are the science concepts presented in the instructional materials accurate and correct? 
[Provide examples of major errors where they are evident. Attach extra page if necessary.] 

1 2 3 4 5 

Substantial, major errors Mostly correct, with Scientifically accurate, 

some minor errors and correct 



c. Do the instructional materials adequately present the major concepts in the standards and 
adequately demonstrate and model the processes of science? 

1 2 3 4 5 

Major concepts and Major concepts and Major concepts and 

processes not addressed processes somewhat addressed processes addressed well 



d. Does the science presented in the instructional materials reflect current disciplinary knowl- 
edge? 

1 2 3 4 5 

The ideas are out of date Somewhat current Current 
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e. Do the instructional materials accurately represent views of science as inquiry as described in 
the National Science Education Standards? 



1 


...2 


3 . . 


4 


5 


Poor examples 




Mixed 




Rich and accurate 


of inquiry 




quality 




examples of inquiry 



f. Do the instructional materials accurately present the history of science? 



1 2 


3 


4 


5 


Poor portrayal 


Mixed 




Rich and accurate 


of history of science 


quality 




portrayal of history of 
science 



g. Do the materials emphasize technology as an area of study? 

1 2 3 4 5 

Little or no Some emphasis Rich and well 

emphasis designed emphasis 



h. Do the materials emphasize the content of earth science? 

1 2 3 4 5 

Little or no Some emphasis Rich and well 

emphasis designed emphasis 



i. Do the materials emphasize the content of physical science? 

1 2 3 4 5 

Little or no Some emphasis Rich and well 

emphasis designed emphasis 
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j. Do the instructional materials provide sufficient activities for students to develop a good understand- 
ing of key science concepts? 

1 2 3 4 5 

Too few learning Activities provide Activities provide 



k. Do the instructional materials provide sufficient opportunities for students to apply their understand- 
ing of the concepts (i.e., designing of solutions to problems or issues)? 

1 2 3 4 5 

Very few Some Very rich in 

application activities application activities application activities 



1. Do the instructional materials present an accurate picture of the nature of science as a dynamic 
endeavor? 



1 2 

The image of science is 
out-of-date, inaccurate, 
or non-existent. 



. .3 4 

The image of 
science is of 
mixed quality. 



5 

The image of 
science is current 
and accurate. 



m. Do the materials develop an appropriate breadth and depth of science content? 

1 2 3 4 5 

Too narrow Somewhat Good balance of 

or too broad balanced breadth and depth 



n. What is the overall quality of the science presented in the instructional materials? 

1 2 3 4 5 

Low Medium High 
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III. The Pedagogical Design 



a. Do the instructional materials provide a logical progression for developing conceptual under- 
standing in science? 



1 

No logical 
progression 
of ideas 



.2 



3 

Somewhat logical 
progression of ideas 



Logical progression 
of ideas that builds 
conceptual understanding 



b. Do the instructional materials provide students the opportunity to make conjectures, gather 
evidence, and develop arguments to support, reject, and revise their preconceptions and explana- 
tions for natural phenomena? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 



c. To what extent do the instructional materials engage students in doing science inquiry? 



1 2 

Very few or very 
contrived activities 
for students to do 
science inquiry 



3 

Some good activities 
for students to do 
science inquiry 



Many rich and authentic 
opportunities for 
students to do 
science inquiry 



d. To what extent do the instructional materials engage students in doing technology problem 
solving? 



1 2 

Very few or very 
contrived activities 
for students to do 
technology 
problem solving 



3 

Some good activities 
for students to do 
technology 
problem solving 



Many rich and authentic 

opportunities 

for students to do 

technology 

problem solving 
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e. To what extent does the curriculum engage students in activities that help them connect science to 
everyday issues and events? 



1 2 . 

Very few or very 
contrived activities 
for students 
to make connections 



3 

Some good activities 
for students to make 
connections 



.4 5 

Many rich and authentic 
opportunities 
for students 
to make connections 



f. How would you rate the overall developmental appropriateness of the instructional materials, 
given its intended audience of ALL students at the targeted level(s)? 

1 2 3 4 5 

Not developmentally Somewhat developmentally Developmentally 

appropriate appropriate appropriate 



g. Do the materials reflect current knowledge about effective teaching and learning practices 
(e.g., active learning, inquiry, community of learners) based on research related to science educa- 
tion? 



1 2 . 

Do not reflect 

current knowledge 

about teaching and learning 



3 4 5 

Somewhat reflective Reflect well 

of current knowledge current knowledge 

about teaching and learning about teaching and 

learning 



h. Do the instructional materials provide students the opportunity to clarify, refine, and consoli- 
date their ideas, and to communicate them through multiple modes? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 

. 1 09 
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i. Do the instructional materials provide students the opportunity to think and communicate 
scientifically? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 



j. Do the instructional materials provide students with activities connecting science with other 
subject areas? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 



k. Are the instructional materials likely to be interesting, engaging, and effective for students? 

1 2 3 4 5 

Not at all interesting Somewhat interesting Interesting and engaging 



1. Are the instructional materials likely to be interesting, engaging, and effective for girls and 
boys? 

1 2 3 4 5 

No sensitivity Some sensitivity Sensitive to 

to gender issues to gender issues gender issues 



m. Are the instructional materials likely to be interesting, engaging, and effective for 
underrepresented and underserved students (e.g., gender, ethnic, urban, rural, with disabilities)? 



1 2 

No sensitivity 
to underrepresented and 
underserved students 



3 

Some sensitivity 
to underrepresented and 
underserved students 



Sensitive to 
underrepresented and 
underserved students 
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n. Does assessment have explicit purposes connected to decisions to be made by teachers (e.g., 
prior knowledge, conceptual understanding, grades)? 



1 

Unclear purposes 


. . .2 3 4 5 

Somewhat clear Clear statement 

purposes of purposes 



o. Do assessments focus on the curriculum’s important content and skills? 



i 

Poor correspondence 


...2 3 4 5 

Fair correspondence Full correspondence 



p. Do the instructional materials include multiple kinds of assessments (e.g., performance, 
paper/pencil, portfolios, student interviews, embedded, projects)? 



i 

Little or no student 
assessment provided 


. . .2 3 4 5 

Some variety of Complete 

student assessment student assessment 

package 



q. Are the assessment practices fair to all students? 



i 

Fair for a few 


...2 3 4 5 

Fair to most Fair to all 

in 
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r. Do the instructional materials include adequate and appropriate uses of a variety of educa- 
tional technologies (e.g., calculators, video, computers, telecommunications)? 



1 2 

Little or no 

educational technology 
included 



3 4 

Some appropriate 
educational technology 
included 



5 

Many appropriate 
applications of 
educational technology 
included 



s. What is the overall quality of the pedagogical design of these instructional materials? 

1 2 3 4 5 

Low Medium High 



t. To what extent are the purposes of the materials clear to students? 

1 2 3 4 5 

Purposes are unclear Purposes are somewhat clear Purposes are clear 
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IV. Implementation and System Support 



a. Will the teachers find the materials interesting and engaging? 

1 2 3 4 5 

Dry and boring Somewhat interesting Interesting and engaging 

and engaging 



b. Do the instructional materials include information and guidance to assist the teacher in imple- 
menting the lessons? 



1 


...2 


3 


...A.... 


5 


No teacher support 




Some teacher support 




Rich and useful teacher 
support 



c. Do the instructional materials provide information about the kind of resources and support 
system required to facilitate the district implementation of the science materials? 

1 2 3 4 5 

No materials support Some materials support Rich and useful materials 

support 



d. Do the instructional materials provide information about how to establish a safe science 
learning environment? 

1 2 3 .4 5 

No safety information Some safety information Rich and useful safety 

information 
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e. Do the instructional materials provide information about the kinds of professional develop- 
ment experiences needed by teachers to implement the materials? 

1 2 3 4 5 

Little or no Partial information Rich and useful 

information provided provided information provided 



f. Do the materials provide guidance in how to link the materials with the district and state 
assessment frameworks and programs? 

1 2 3 4 5 

No guidance Some guidance Rich and useful guidance 



g. Do the materials provide guidance and assistance for involving administrators, parents, and 
the community at large actively in supporting school science? 

1 2 3 4 5 

No guidance Some guidance Rich and useful guidance 



h. Overall, are the materials usable by, realistic in expectations of, and supportive of teachers? 
1 2 3 4 5 

Teacher unfriendly Somewhat teacher friendly Teacher friendly 
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V. Major Strengths and Weaknesses 

a. In your opinion, what are the three major strengths of this curriculum? 



b. In your opinion, what are the three major weaknesses of this curriculum? 



11 5 
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VI. Overall Quality, Value, and Contribution 

a. In your opinion, what is the overall quality of these materials relative to: 





low 








high 


- turning students on to science? 


1 


.2... 


...3... 




5 


- making students think? 


1 


.2... 




...4... 


5 


- quality of science content? 


1 


.2... 




...4... 


5 


- quality of pedagogy? 


1 


.2... 


...3... 


...4... 


5 


- quality of classroom assessments? 


1 


.2... 






5 


- encouraging teachers to teach differently? 


1 


.2'... 


...3... 


...4... 


5 



b. In your opinion, what is the overall quality of these instructional materials? 

1 2 3 A 5 

Low Medium High 



c. To what extent would you encourage the dissemination, adoption, and implementation of this 
curriculum? 

1 2 3 4 5 

Not worthy of OK to disseminate, OK to disseminate, 

dissemination, adoption, adopt, and implement adopt, and implement 

nor implementation if revised as is 
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Guidelines for Section II. 



The following is a brief outline of the National Science Education Standards. It should be used 
to guide your responses to Section II. 



CONTENT STANDARD 


A. Science as Inauirv 


1. Abilities necessary to do science inauirv 


2. Understandings about scientific inauirv 


B. Physical Science 


1. Properties of objects and materials 


2. Position and motion of objects 


3. Light, heat, electricity, and magnetism 


C. Life Science 


1 . The characteristics of organisms 


2. Life cycles of organisms 


3. Organisms and environments 


D. Earth and Space Science 


1 . Properties of Earth materials 


2. Objects in the skv 


E. Science and Technology 


1. Abilities to distinguish between natural objects and objects made bv humans 


2. Abilities of technological design 


3. Understandings about science and technology 


F. Science in Personal and Social Perspectives 


1. Personal health 


2. Characteristics and changes in populations 


3. Types of resources 


4. Changes in environments 


5. Science and technology in local challenges 


G. History and Nature of Science 


1 . Science as a human endeavor 


H. Unifying concepts and processes 


1 . Order and organization 


2. Evidence, models, and explanation 


3. Change, constancy, and measurement 


4. Evolution and eauilibrium 


5. Form and function 
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Framework for Review 

Instructional Materials for High School Science 1 



Title: 

Author(s): 

Publisher: Copyright date: 

Reviewed by: Date: 

I. Descriptors 

a. Write a brief description of the components of the curriculum upon which this review is based 
(e.g., teachers guide, student books, hands-on materials, multimedia material). That is, what 
materials did you receive and include in your review? 



b. Write a brief description of the purpose and broad goals of these materials. That is, what were 
the stated purposes and what were the actual purposes of the materials? 



c. What grade levels do the materials serve? 

9 10 11 12 



d. Are the instructional materials designed to 

provide a complete multi-year program for high school science. 

provide a complete one-year course for high school science. 

provide multiple modules or units that could be used to supplement other course materials 

for high school science. 

provide a single module or collection of activities that could be used to supplement other 

course materials for high school science. 

other (explain): 



' NOTE: This framework is adapted from an instrument developed by Inverness Research under contract to the 
National Science Foundation. The framework was refined as part of a panel review of NSF-supported materials for 
middle school science, which was limited to projects that provide jtr a y ear ~l° n 8 course of study. 
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e. What are the major domains/topics of content covered by these materials? 
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II. Quality of the Science 



Directions: For each item, circle the number corresponding with your response to the question. 

Write an explanation for your rating of each item below the item. 

a. Does the content in the instructional materials align well with all eight areas of the Content 
Standards as described in the National Science Education Standards (NSES)? 

(See attached guidelines) 



1 2 .... 

Omits substantial content 
included in NSES 
and/or includes substantial content 
not recommended in NSES 



3 

Some misalignment 
of content with 
recommendations 
in NSES 



The curriculum 
aligns well with 
content recommendations 
in NSES 



b. Are the science concepts presented in the instructional materials accurate and correct? 
[Provide examples of major errors where they are evident. Attach extra page if necessary.] 

1 2 3 4 5 

Substantial, major errors Mostly correct, with Scientifically accurate, 

some minor errors and correct 



c. Do the instructional materials adequately present the major concepts in the standards and 
adequately demonstrate and model the processes of science? 

1 2 3 4 5 

Major concepts and Major concepts and Major concepts and 

processes not addressed processes somewhat addressed processes addressed well 



d. Does the science presented in the instructional materials reflect current disciplinary 
knowledge? 

1 2 3 4 5 

The ideas are out of date Somewhat current Current 
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e. Do the instructional materials accurately represent views of science as inquiry as described in 
the National Science Education Standards? 

1 .2 3 4 5 

Poor implies Mixed Rich and accurate 

of inquiry quality examples of inquiry 



f. Do the instructional materials accurately present the history of science? 

1 2 3 4 5 

Poor portrayal Mixed Rich and accurate 

of history of science quality portrayal of history of 

science 



g. Do the materials emphasize technology as an area of study? 
1 2 3 4 . 

Little or no Some emphasis 

emphasis 



5 

Rich and well 
designed emphasis 



h. Do the materials emphasize the personal and societal dimensions of science? 

1 2 3 4 5 

Little or no Some emphasis Rich and well 

emphasis designed emphasis 
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i. Do the materials emphasize the content of life science? 



1 2 

Little or no 
emphasis 



. .3 4 

Some emphasis 



5 

Rich and well 
designed emphasis 



j. Do the materials emphasize the content of earth science? 

1 2 3 4 5 

Little or no Some emphasis Rich and well 

emphasis designed emphasis 



k. Do the materials emphasize the content of physical science? 

1 2 3 4 5 

Little or no Some emphasis Rich and well 

emphasis designed emphasis 



1. Do the instructional materials provide sufficient activities for students to develop a good 
understanding of key science concepts? 



1 


...2 


3 4 


5 


Too few learning 




Activities provide 


Activities provide 


activities 




some opportunity 


many rich opportunities 






for students to learn 


to learn key science 






some important concepts 


concepts 



m. Do the instructional materials provide sufficient opportunities for students to apply their 
understanding of the concepts (i.e., designing of solutions to problems or issues)? 

1 2 3 4 5 

Very few Some Very rich in 

application activities application activities application activities 
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n. Do the instructional materials present an accurate picture of the nature of science as a dy- 
namic endeavor? 



1 2 

The image of science is 
out-of-date, inaccurate, 
or non-existent 



. .3 4 

The image of 
science is of 
mixed quality 



5 

The image of 
science is current 
and accurate 



o. Do the materials develop an appropriate breadth and depth of science content? 

1 2 3 4 5 

Too narrow Somewhat Good balance of 

or too broad balanced breadth and depth 



p. What is the overall quality of the science presented in the instructional materials? 
1 2 3 4 5 

Low Medium High 
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III. The Pedagogical Design 



a. Do the instructional materials provide a logical progression for developing conceptual under- 
standing in science? 



1 

No logical 
progression 
of ideas 



2 , 



3 

Somewhat logical 
progression of ideas 



Logical progression 
of ideas that builds 
conceptual understanding 



b. Do the instructional materials provide students the opportunity to make conjectures, gather 
evidence, and develop arguments to support, reject, and revise their preconceptions and explana- 
tions for natural phenomena? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 



c. To what extent do the instructional materials engage students in doing science inquiry? 



1 2 . 

Very few or very 
contrived activities 
for students to do 
science inquiry 



3 

Some good activities 
for students to do 
science inquiry 



Many rich and authentic 
opportunities for 
students to do 
science inquiry 



d. To what extent do the instructional materials engage students in doing technology problem 
solving? 



1 

Very few or very 
contrived activities 
for students to do 
technology 
problem solving 



O 

cd i r 



2 , 



3 

Some good activities 
for students to do 
technology 
problem solving 

12 5 



Many rich and authentic 

opportunities 

for students to do 

technology 

problem solving 
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e. To what extent does the curriculum engage students in activities that help them connect 
science to everyday issues and events? 



1 2 . 

Very few or very 
contrived activities 
for students 
to make connections 



3 

Some good activities 
for students to make 
connections 



.4 5 

Many rich and authentic 
opportunities 
for students 
to make connections 



f. How would you rate the overall developmental appropriateness of the instructional materials, 
given its intended audience of ALL students at the targeted level(s)? 

1 2 3 4 5 

Not developmental^ Somewhat developmentally Developmentally 

appropriate appropriate appropriate 



g. Do the materials reflect current knowledge about effective teaching and learning practices 
(e.g., active learning, inquiry, community of learners) based on research related to science educa- 
tion? 

1 2 3 4 5 

Do not reflect Somewhat reflective Reflect well 

current knowledge of current knowledge current knowledge 

about teaching and learning about teaching and learning about teaching and 

learning 



h. Do the instructional materials provide students the opportunity to clarify, refine, and consoli- 
date their ideas, and to communicate them through multiple modes? 



1 2 

No opportunity 



O 



3 

Some opportunity 



Rich and well 
designed opportunity 



tee 
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i. Do the instructional materials provide students the opportunity to think and communicate 
scientifically? 



1 

No opportunity 


. .2 3 4 5 

Some opportunity Rich and well 

designed opportunity 



j. Do the instructional materials provide students with activities connecting science with other 
subject areas? 



i 

No opportunity 


. .2 3 4 5 

Some opportunity Rich and well 

designed opportunity 



k. Are the instructional materials likely to be interesting, engaging, and effective for students? 



i 

Not at all interesting 


. .2 3 4 5 

Somewhat interesting Interesting and engaging 



1. Are the instructional materials likely to be interesting, engaging, and effective for girls and 
boys? 



i 

No sensitivity 
to gender issues 


. .2 3 4 5 

Some sensitivity Sensitive to 

to gender issues gender issues 



m. Are the instructional materials likely to be interesting, engaging, and effective for 
underrepresented and underserved students (e.g., gender, ethnic, urban, rural, with disabilities)? 



i 

No sensitivity 
to underrepresented and 
underserved students 


Some sensitivity Sensitive to 

to underrepresented and underrepresented and 

underserved students underserved students 
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n. Does assessment have explicit purposes connected to decisions to be made by teachers (e.g., 
prior knowledge, conceptual understanding, grades)? 

1 2 3 4 5 

Unclear purposes Somewhat clear Clear statement 

purposes of purposes 



o. Do assessments focus on the curriculum’s important content and skills? 

1 2 3 4 5 

Poor correspondence Fair correspondence Full correspondence 



p. Do the instructional materials include multiple kinds of assessments (e.g., performance, 
paper/pencil, portfolios, student interviews, embedded, projects)? 

1 2 3 4 5 

Little or no student Some variety of Complete 

assessment provided student assessment student assessment 

package 



q. Are the assessment practices fair to all students? 

1 2 3 4 5 

Fair for a few Fair to most Fair to all 



0 
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r. Do the instructional materials include adequate and appropriate uses of a variety of educa- 
tional technologies (e.g., calculators, video, computers, telecommunications)? 

1 2 3 4 5 

Little or no Some appropriate Many appropriate 

educational technology educational technology applications of 

included included educational technology 

included 



s. What is the overall quality of the pedagogical design of these instructional materials? 
1 2 3 4 5 

Low Medium High 



t. To what extent are the purposes of the materials clear to students? 

1 2 3 4 5 

Purposes are unclear Purposes are somewhat clear Purposes are clear 



3 
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IV. Implementation and System Support 

a. Will the teachers find the materials interesting and engaging? 

1 , . . . , 2 . 3 4 5 

Dry and boring Somewhat interesting Interesting and engaging 

and engaging 



b. Do the instructional materials include information and guidance to assist the teacher in imple- 
menting the lessons? 

1 2 3 4 5 

No teacher support Some teacher support Rich and useful teacher 

support 



c. Do the instructional materials provide information about the kind of resources and support 
system required to facilitate the district implementation of the science materials? 

1 2 3 4 5 

No materials support Some materials support Rich and useful materials 

support 



d. Do the instructional materials provide information about how to establish a safe science 
learning environment? 

1 2 3 4 5 

No safety information Some safety information Rich and useful safety 

information 



0 
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e. Do the instructional materials provide information about the kinds of professional development 
experiences needed by teachers to implement the materials? 

1 2 3 4 5 

Little or no Partial information Rich and useful 

information provided provided information provided 



f. Do the materials provide guidance in how to link the materials with the district and state 
assessment frameworks and programs? 

1 2 3 4 5 

No guidance Some guidance Rich and useful guidance 



g. Do the materials provide guidance and assistance for involving administrators, parents, and 
the community at large actively in supporting school science? 

1 2 3 4 5 

No guidance Some guidance Rich and useful guidance 



h. Overall, are the materials usable by, realistic in expectations of, and supportive of teachers? 
1 2 3 4 5 

Teacher unfriendly Somewhat teacher friendly Teacher friendly 



O - 
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V. Major Strengths and Weaknesses 

a. In your opinion what are the three major strengths of this curriculum? 



b. In your opinion, what are the three major weaknesses of this curriculum? 
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VI. Overall Quality, Value, and Contribution 

a. In your opinion what is the overall quality of these materials relative to: 

low high 

- turning students on to science? 1 2 3 4 5 

- making students think? 1 2 3 4 5 

- quality of science content? 1 2 3 4 5 

- quality of pedagogy? 1 2 3 4 5 

- quality of classroom assessments? 1 2 3 4 5 

- encouraging teachers to teach differently? 1 2 3 4 5 



b. In your opinion, what is the overall quality of these instructional materials? 

1 2 3 4 5 

Low Medium High 



c. To what extent would you encourage the dissemination, adoption, and implementation of this 
curriculum? 



1 2 

Not worthy of 
dissemination, adoption, 
nor implementation 



3 4. 

OK to disseminate 
adopt, and implement 
if revised 



5 

OK to disseminate, 
adopt, and implement 
as is 
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Use this page for additional notations: 



134 



ERiC 



Science Foundation 



123 



Guidelines for Section II. 



The following is a brief outline of the National Science Education Standards. It should be used 
to guide your responses to Section II. 



CONTENT STANDARD 


Science as Inauirv 


1 . Abilities necessary to do science inauirv 


l. Understandings about scientific inauirv 


B. Physical Science 


1 . Structure of the atom 


l. Structure and properties of matter 


3. Chemical reactions 


Forces and motions 


5. Conservation of energy and increase in disorder 


3. Interactions of energy and matter 


C. Life Science 


1. The cell 


l. The molecular basis of heredity 


3. Biological evolution 


I. The interdependence of organisms 


5. Matter, energy, and organization in living systems 


3. The nervous system and behavior of organisms 


D. Earth and Space Science 


1 . Energy in the Earth system 


l. Geochemical cycles 


3. Origin and evolution of the Earth 


I. Origin and evolution of the universe 


E. Science and Technology 


l. Abilities of technological design 


2. Understandings about science and technology 


E. Science in Personal and Social Perspectives 


1 . Personal and community health 


l. Population growth 


3. Natural resources 


I. Environmental aualitv 


5. Natural and human-induced hazards 


5. Science and technology in local, national, and global challenges 


G. History and Nature of Science 


l. Science as a human endeavor 


l. Nature of scientific knowledge 


3. Historical perspectives 
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H. Unifying concepts and processes 

I . Systems, order and organization 

l. Evidence, models, and explanation 
3. Change, constancy, and measurement 

Evolution and equilibrium 

5. Form and function 
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Framework for Review 

Instructional Materials for Elementary School Mathematics 1 



Title: 

Author(s): 

Publisher: Copyright date: 

Reviewed by: Date: 

I. Descriptors 

a. Write a brief description of the components of the curriculum upon which this review is based 
(e.g., teachers guide, student books, hands-on materials, multimedia material). That is, what 
materials did you receive and include in your review? 



b. Write a brief description of the purpose and broad goals of these materials. That is, what were 
the stated purposes and what were the actual purposes of the materials? 



c. What grade levels do the materials serve? 

K 1 2 3 4 5 



d. Are the instructional materials designed to 

provide a complete multi-year program for elementary school mathematics. 

provide a complete one-year course for elementary school mathematics. 

provide multiple modules or units that could be used to supplement other course materials 

for elementary school mathematics. 

provide a single module or collection of activities that could be used to supplement other 

course materials for elementary school mathematics. 

other (explain): 



' NOTE: This framework is adapted from an instrument developed by Inverness Research under contract to the 
National Science Foundation. The framework was refined as part of a panel review of NSF-supported materials for 
middle school science, which was limited td projects that provide at least a year-long course of study. 
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e. What are the major domains/topics of content covered by these materials? 



O 
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II. Quality of the Mathematics 



Directions: For each item, circle the number corresponding with your response to the question. 
Write an explanation for your rating of each item below the item. 

a. Does the content in the instructional materials align well with all thirteen areas of the Curricu- 
lum Standards as described in the National Council of Teachers of Mathematics Curriculum and 
Evaluation Standards for School Mathematics (NCTM)? 

(See attached guidelines) 



1 2 . . . . 

Omits substantial content 
included in NCTM 
and/or includes substantial content 
not recommended in NCTM 



3 

Some misalignment 
of content with 
recommendations 
in NCTM 



The curriculum 
aligns well with 
content recommendations 
in NCTM 



b. Are the mathematics concepts presented in the instructional materials accurate and correct? 
[Provide examples of major errors where they are evident. Attach extra page if necessary.] 

1 2 3 4 5 

Substantial, major errors Mostly correct, with Mathematically accurate, 

some minor errors and correct 



c. Do the instructional materials adequately present the major concepts and adequately demon- 
strate and model the processes of mathematics? 

1 2 3 4 5 

Major concepts and Major concepts and Major concepts and 

processes not addressed processes somewhat addressed processes addressed well 



d. Do the instructional materials accurately represent views of mathematical problem solving as 
described in the NCTM Curriculum and Evaluation Standards for School Mathematics? 

1 2 3 4 5 

Poor portrayal Mixed Rich and accurate 

of problem solving quality portrayal of 

problem solving 
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e. Do the materials use technology as a tool for learning mathematics? 

1 2 3 4 5 

Little or no Some emphasis Rich and well 

use designed use 



f. Do the materials emphasize communication about mathematics through a variety of modali- 
ties? 

1 2 3 4 5 

Little or no Some emphasis, Rich and well 

emphasis, few some modalities designed emphasis, 

modalities varied modalities 



g. Do the materials appropriately address mathematical reasoning? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



h. Do the materials appropriately address computation? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



i. Do the materials appropriately address estimation? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



•> I 
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j. Do the materials appropriately address number sense and numeration? 



1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



k. Do the materials appropriately address concepts of whole number operation? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



l. Do the materials appropriately address whole number computation? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 

m. Do the materials appropriately address geometry and spatial sense? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



n. Do the materials appropriately address measurement? 

1 2 .3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



3 
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o. Do the materials appropriately address statistics and probability? 



1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



p. Do the materials emphasize fractions and decimals? 

5 

Rich and well 
designed emphasis 



Little or no Some emphasis 

emphasis 



q. Do the instructional materials provide sufficient activities for students to develop a good 
understanding of key mathematics concepts? 



1 


. . . .2 


3 


..A.... 


5 


Too few learning 




Activities provide 




Activities provide 


activities 




some opportunity 




many rich opportunities 






for students to learn 




to learn key mathematics 






some important concepts 




concepts 



r. Do the instructional materials provide sufficient opportunities for students to apply their 
understanding of the concepts (i.e., designing of solutions to problems or issues)? 

1 2 3 4 5 

Very few Some Very rich in 

application activities application activities application activities 



s. Do the materials develop an appropriate breadth and depth of mathematics content? 

1 2 3 4 5 

Too narrow Somewhat Good balance of 

or too broad balanced breadth and depth 
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t. What is the overall quality of the mathematics presented in the instructional materials? 



1 2 3 4 5 

Low Medium High 
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III. The Pedagogical Design 



a. Do the instructional materials provide a logical progression for developing conceptual under- 
standing in mathematics? 



1 

No logical 
progression 
of ideas 



2 , 



3 

Somewhat logical 
progression of ideas 



Logical progression 
of ideas that builds 
conceptual understanding 



b. Do the instructional materials provide students the opportunity to formulate, solve, and reflect 
critically on problems? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 



c. To what extent are the mathematical concepts embedded in learner-appropriate 






1 2 


3 4 


5 


Very few or very 


Some good activities 


Many rich and authentic 


contrived activities 


for students to do 


opportunities for 


for students to do 
mathematical problem solving 


mathematical problem solving 


students to do 
mathematical 
problem solving 


d. How would you rate the overall developmental appropriateness of the instructional materials, 
given its intended audience of ALL students at the targeted level(s)? 


1 2 


3 4 


5 


Not developmental^ 


Somewhat developmental^ 


Developmental^ 


appropriate 


appropriate 


appropriate 
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e. Do the materials reflect current (that is, within the last 5 years) knowledge about effective 
teaching and learning practices (e.g., active learning, inquiry, community of learners) based on 
research related to mathematics education? 



1 2 

Do not reflect 

current knowledge 

about teaching and learning 



3 4 

Somewhat reflective 
of current knowledge 
about teaching and learning 



. .5 

Reflect well 
current knowledge 
about teaching and 
learning 



f. Do the instructional materials provide students the opportunity to clarify, refine, and consolidate their 
ideas? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 



g. Do the instructional materials provide students with activities connecting mathematics with other subject 
areas? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 



h. Are the instructional materials likely to be interesting, engaging, and effective for girls and boys? 



1 2 

No sensitivity 
to gender issues 



3 .4 

Some sensitivity 
to gender issues 



. .5 

Sensitive to 
gender issues 



> f 
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i. Are the instructional materials likely to be interesting, engaging, and effective for underrepresented and 
underserved students (e.g., ethnic, urban, rural, with disabilities)? > 



1 2 , 

No sensitivity 
to underrepresented and 
and underserved students 



3 4 

Some sensitivity 
to underrepresented and 
underserved students 



. .5 

Sensitive to 
underrepresented 
underserved students 



j. Does assessment have explicit purposes connected to decisions to be made by teachers (e.g., prior 
knowledge, conceptual understanding, grades)? 

1 2 3 4 5 

Unclear purposes Somewhat clear Clear statement 

purposes of purposes 



k. Do assessments focus on the curriculum’s important content and skills? 

1 2 3 4 5 

Poor correspondence Fair correspondence Full correspondence 



1. Do the instructional materials include multiple kinds of assessments (e.g., performance, paper/pencil, 
portfolios, student interviews, embedded, projects)? 

1 2 3 4 5 

Little or no student Some variety of Complete 

assessment provided student assessment student assessment 

package 
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m. Are the assessment practices fair to all students? 



4 



1 2 3 . . . 

Fair for a few Fair to most 



. .5 

Fair to all 



n. Do the instructional materials include adequate and appropriate uses of a variety of educa- 
tional technologies (e.g., calculators, video, computers, telecommunications)? 



1 2 

Little or no 

educational technology 
included 



3 4 

Some appropriate 
educational technology 
included 



. .5 

Many appropriate 
applications of 
educational 
technology included 



o. What is the overall quality of the pedagogical design of these instructional materials? 
1 2 3 4 5 

Low Medium High 
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IV. Implementation and System Support 



a. Will the teachers find the materials interesting and engaging? 

1 2 ....3 4 .5 

Dry and boring Somewhat interesting Interesting and engaging 

and engaging 



b. Do the instructional materials include information and guidance to assist the teacher in imple- 
menting the lessons? 

1 2 3 4 5 

No teacher support Some teacher support Rich and useful teacher 

support 



c. Do the instructional materials provide information about the kind of resources and support 
system required to facilitate the district implementation of the required mathematics materials? 

1 2 3 4 5 

No materials support Some materials support Rich and useful materials 

support 



d. Do the instructional materials provide information about the kinds of professional develop- 
ment experiences needed by teachers to implement the materials? 

1 2 3 4 5 

Little or no Partial information Rich and useful 

information provided provided information provided 
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e. Do the materials provide guidance in how to link the materials with the district and state assessment 
frameworks and programs? 

1 2 3 4 5 

No guidance Some guidance Rich and useful guidance 

f. Do the materials provide guidance and assistance for involving administrators, parents, and the 
community at large actively in supporting school mathematics? 

1 2 3 4 5 

No guidance Some guidance Rich and useful guidance 



g. Overall, are the materials usable by, realistic in expectations of, and supportive of teachers? 
1 2 3 4 ... 5 

Teacher unfriendly Somewhat teacher friendly Teacher friendly 
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V. Major Strengths and Weaknesses 

a. In your opinion, what are the three major strengths of this curriculum? 



b. In your opinion, what are the three major weaknesses of this curriculum? 
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VI. Overall Quality, Value, and Contribution 

a. In your opinion, what is the overall quality of these materials relative to: 

low high 

- turning students on to mathematics? 1 2 3 4 5 

- making students think? 1 2 3 4 5 

- quality of mathematics content? 1 2.. .....3 4 5 

- quality of pedagogy? 1 2 3 4 5 

- quality of classroom assessments? 1 2 3 4 5 

- encouraging teachers to teach differently? 1 ..2 3 4 5 

b. In your opinion, what is the overall quality of these instructional materials? 

1 2 3 4 5 

Low Medium High 

c. To what extent would you encourage the dissemination, adoption, and implementation of this 
curriculum? 

1 2 3 4. 5 

Not worthy of OK to disseminate, OK to disseminate, 

dissemination, adoption, adopt, and implement adopt, and implement 

nor implementation if revised as is 
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Guidelines for Section II. 

The following is a brief outline of the National Council of Teachers of Mathematics Curriculum and 
Evaluation Standards for School Mathematics. It should be used to guide your responses to Section 



CONTENT STANDARD 


A. Mathematics as Problem Solving 


1. Use problem solving approaches to investigate and understand mathematical content 


2. Formulate problems from evervdav and mathematical situations 


3. Develop and applv strategies to solve a wide variety of problems 


4. Verify and interpret results with respect to the original problem 


5. Acauire confidence in using mathematics meaningfully 


B. Mathematics as Communication 


1. Relate physical materials, pictures, and diagrams to mathematical ideas 


2. Reflect on and clarify their thinking about mathematical ideas and situations 


3. Relate their evervdav language to mathematical language and symbols 


4. Realize that representing, discussing, reading, writing, and listening to mathematics 
are a vital part of learning and using mathematics 


C. Mathematics as Reasoning 


1. Draw logical conclusions about mathematics 


2. Use models, known facts, properties, and relationships to explain their thinking 


3. Justify their answers and solution processes 


4. Use patterns and relationships to analyze mathematical situations 


5. Believe that mathematics makes sense 


D. Mathematical Connections 


1. Link conceptual and procedural knowledge 


2. Relate various representations of concepts or procedures to one another 


3. Recognize relationships among different topics in mathematics 


4. Use mathematics in other curriculum areas 


5. Use mathematics in their daily lives 


E. Estimation 


1 . Explore estimation strategies 


2. Recognize when an estimate is appropriate 


3. Determine the reasonableness of results 


4. Apply estimation in working with quantities, measurement, computation, and 
problem solving 


F. Number Sense and Numeration 


1 . Construct number meanings through real-world experiences and the use of physical 
materials 


2. Understand our numeration system by relating counting, grouping, and place-value 
concepts 


3. Develop number sense 


4. Interpret the multiple uses of numbers encountered in the real world 
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CONTENT STANDARD 
(Continued) 


G. Concepts of Whole Number Operations 


1. Develop meaning for the operations by modeling and discussing a rich variety of 
problem situations 


2. Relate the mathematical language and symbolism of operations to problem situations 
and informal language 


3. Recognize that a wide variety of problem structures can be represented by a single 
operation 


4. Develop operation sense 


H. Whole Number Computation 


1. Model, explain, and develop reasonable proficiency with basic facts and algorithms 


2. Use a variety of mental computation and estimation techniques 


3. Use calculators in appropriate computational situations 


4. Select and use computation techniques appropriate to specific problems and determine 
whether the results are reasonable 


I. Geometry and Spatial Sense 


1. Describe, model, draw, and classify shapes 


2. Investigate and predict the results of combining, subdividing, and changing shapes 


3. Develop spatial sense 


4. Relate geometric ideas to number and measurement ideas 


5, Recognize and appreciate geometry in their world 


J. Measurement 


1. Understand the attributes of length, capacity, weight, mass, area, volume, time, 
temperature, and angle 


2. Develop the process of measuring and concepts related to units of measurement 


3. Make and use estimates of measurement 


4. Make and use measurements in problem and everyday situations 


K. Statistics and Probability 


1 . Collect, organize, and describe data 


2. Construct, read, and interpret displays of data 


3. Formulate and solve problems that involve collecting and analyzing data 


4. Explore concepts of chance 


L. Fractions and Decimals 


1. Develop concepts of fractions, mixed numbers, and decimals 


2. Develop number sense for fractions and decimals 


3. Use models to relate fractions to decimals and to find equivalent fractions 


4. Use models to explore operations on fractions and decimals 


5. Apply fractions and decimals to problem situations 


M. Patterns and Relationships 


1. Recognize, describe, extend, and create a wide variety of patterns 


2. Represent and describe mathematical relationships 


3. Explore the use of variables and open sentences to express relationships 
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Framework for Review 

Instructional Materials for Middle School Mathematics 1 



Title: 

Author(s): 

Publisher: Copyright date: 

Reviewed by: Date: 

I. Descriptors 

a. Write a brief description of the components of the curriculum upon which this review is based 
(e.g., teachers guide, student books, hands-on materials, multimedia material). That is, what 
materials did you receive and include in your review? 



b. Write a brief description of the purpose and broad goals of these materials. That is, what were 
the stated purposes and what were the actual purposes of the materials? 



c. What grade levels do the materials serve? 

5 6 7 8 



d. Are the instructional materials designed to 

provide a complete multi-year program for middle school mathematics. 

provide a complete one-year course for middle school mathematics. 

provide multiple modules or units that could be used to supplement other course materials 

for middle school mathematics. 

provide a single module or collection of activities that could be used to supplement other 

course materials for middle school mathematics. 

other (explain): 



' NOTE: This framework is adapted from an instrument developed by Inverness Research under contract to the 
National Science Foundation. The framework was refined as part of a panel review of NSF-supported materials for 
middle school science, which was limited to projects that provide at least a year-long course of study. 
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e. What are the major domains/topics of content covered by these materials? 
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II. Quality of the Mathematics 



Directions: For each item, circle the number corresponding with your response to the question. 

Write an explanation for your rating of each item below the item. 



a. Does the content in the instructional materials align well with all thirteen areas of the Curricu- 
lum Standards as described in the National Council of Teachers of Mathematics Curriculum and 
Evaluation Standards for School Mathematics (NCTM)? 

(See attached guidelines) 



1 2 3 

Omits substantial content Some misalignment 

included in NCTM of content with 

and/or includes substantial content recommendations 

not recommended in NCTM in NCTM 



The curriculum 
aligns well with 
content recommendations 
in NCTM 



b. Are the mathematics concepts presented in the instructional materials accurate and correct? 
[Provide examples of major errors where they are evident. Attach extra page if necessary] 

1 2 3 4 5 

Substantial, major errors Mostly correct, with Mathematically accurate, 

some minor errors and correct 



c. Do the instructional materials adequately present the major concepts and adequately demon- 
strate and model the processes of mathematics? 

1 2 3 4 5 

Major concepts and Major concepts and Major concepts and 

processes not addressed processes somewhat addressed processes addressed well 
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d. Do the instructional materials accurately represent views of mathematical problem solving as de- 
scribed in the NCTM Curriculum and Evaluation Standards for School Mathematics? 



1 2 , 

Poor portrayal 
of problem solving 



3 

Mixed 

quality 



Rich and accurate 
portrayal of 
problem solving 



e. Do the materials use technology as a tool for learning mathematics? 

1 2 3 4 5 

Little or no Some emphasis Rich and well 

use designed use 



f. Do the materials emphasize communication about mathematics through a variety of modali- 
ties? 



1 2 

Little or no 
emphasis, few 
modalities 



3 

Some emphasis, 
some modalities 



Rich and well 
designed emphasis, 
varied modalities 



g. Do the materials appropriately address mathematical reasoning? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 
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h. Do the materials appropriately address computation? 



1 2 . 


3 4 5 


Not appropriately addressed 


Somewhat appropriately addressed Appropriately addressed 



i. Do the materials appropriately address estimation? 



1 2 . 


3 4 5 


Not appropriately addressed 


Somewhat appropriately addressed Appropriately addressed 



j. Do the materials appropriately address number and number relationships? 



1 2 . 


3 4 5 


Not appropriately addressed 


Somewhat appropriately addressed Appropriately addressed 



k. Do the materials appropriately address number systems and number theory? 



1 2 . 


3 4 5 


Not appropriately addressed 


Somewhat appropriately addressed Appropriately addressed 
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1. Do the materials appropriately address patterns? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



m. Do the materials appropriately address functions? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



n. Do the materials appropriately address algebra? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



o. Do the materials appropriately address geometry? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 
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p. Do the materials appropriately address measurement? 



1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



q. Do the materials appropriately address statistics? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



r. Do the materials appropriately address probability? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



s. Do the instructional materials provide sufficient activities for students to develop a good 
understanding of key mathematics concepts? 



1 


...2 


3 






Too few learning 




Activities provide 




Activities provide 


activities 




some opportunity 




many rich opportunities 






for students to learn 




to learn key mathematics 






some important concepts 




concepts 



O 
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t. Do the instructional materials provide sufficient opportunities for students to apply their 
understanding of the concepts (i.e., designing of solutions to problems or issues)? 

1 2 3 4 5 

Very few Some Very rich in 

application activities application activities application 

activities 



u. Do the materials develop an 

1 2 

Too narrow 
or too broad 



appropriate breadth and depth 

3 

Somewhat 

balanced 



* mathematics content? 

.4 5 

Good balance of 
breadth and 
depth 



v. What is the overall quality of the mathematics presented in the instructional materials? 
1 2 3 4 5 

Low Medium High 
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HI. The Pedagogical Design 



a. Do the instructional materials provide a logical progression for developing conceptual under- 
standing in mathematics? 



1 

No logical 
progression 
of ideas 



2 , 



3 

Somewhat logical 
progression of ideas 



.4 



5 

Logical progression 
of ideas that builds 
conceptual understanding 



b. Do the instructional materials provide students the opportunity to formulate, solve, and reflect 
critically on problems? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 



c. To what extent are the mathematical concepts embedded in learner-appropriate contexts? 



1 2 . 

Very few or very 
contrived activities 
for students to do 
mathematical problem solving 



3 

Some good activities 
for students to do 
mathematical problem solving 



.4 5 

Many rich and authentic 
opportunities for 
students to do 
mathematical 
problem solving 



d. How would you rate the overall developmental appropriateness of the instructional materials, 
given its intended audience of ALL students at the targeted level(s)? 

1 2 3 4 5 

Not developmentally Somewhat developmentally Developmentally 

appropriate appropriate i ^ appropriate 

- io3 
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e. Do the materials reflect current (that is, within the last 5 years) knowledge about effective teaching 
and learning practices (e.g., active learning, inquiry, community of learners) based on research related to 
mathematics education? 

1 .2 3 4 5 

Do not reflect Somewhat reflective Reflect well 

current knowledge of current knowledge current knowledge 

about teaching and learning about teaching and learning about teaching 

and learning 



f. Do the instructional materials provide students the opportunity to clarify, refine, and consolidate their 
ideas? 



1 


2 


3 


4 


5 


No opportunity 




Some opportunity 




Rich and well 

designed 

opportunity 



g. Do the instructional materials provide students with activities connecting mathematics with other 
subject areas? 



1 


2 . . 


3 


4 


5 


No opportunity 




Some opportunity 




Rich and well 

designed 

opportunity 



h. Are the instructional materials likely to be interesting, engaging, and effective for girls and boys? 



1 2 3 4 

No sensitivity Some sensitivity 

to gender issues to gender issues 



. .5 

Sensitive to 
gender issues 
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i. Are the instructional materials likely to be interesting, engaging, and effective for underrepresented and 
underserved students (e.g., ethnic, urban, rural, with disabilities)? 



1 2 

No sensitivity 
to underrepresented and 
underserved students 



3 4 

Some sensitivity 
to underrepresented and 
underserved students 



. .5 

Sensitive to 
underrepresented 
and underserved 
students 



j. Does assessment have explicit purposes connected to decisions to be made by teachers (e.g., prior 
knowledge, conceptual understanding, grades)? 

1 2 3 4 5 

Unclear purposes Somewhat clear Clear statement 

purposes of purposes 



k. Do assessments focus on the curriculum’s important content and skills? 

1 2 3 4 5 

Poor correspondence Fair correspondence Full correspondence 



1. Do the instructional materials include multiple kinds of assessments (e.g., performance, paper/pencil, 
portfolios, student interviews, embedded, projects)? 

1 2 3 4 5 

Little or no student Some variety of Complete 

assessment provided student assessment student assessment 

package 
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m. Are the assessment practices fair to all students? 



,4 



1 2 3 . . . 

Fair for a few Fair to most 



. .5 

Fair to all 



n. Do the instructional materials include adequate and appropriate uses of a variety of educational tech- 
nologies (e.g., calculators, video, computers, telecommunications)? 



1 2 

Little or no 

educational technology 
included 



3 4 

Some appropriate 
educational technology 
included 



. .5 

Many appropriate 
applications of 
educational 
technology included 



o. What is the overall quality of the pedagogical design of these instructional materials? 

1 2 3 4 5 

Low Medium High 
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IV. Implementation and System Support 



a. Will the teachers find the materials interesting and engaging? 

1 2 3 4 5 

Dry and boring Somewhat interesting Interesting and engaging 

and engaging 



b. Do the instructional materials include information and guidance to assist the teacher in imple- 
menting the lessons? 

1 2 3 4 5 

No teacher support Some teacher support Rich and useful teacher 

support 



c. Do the instructional materials provide information about the kind of resources and support 
system required to facilitate the district implementation of the required mathematics materials? 

1 2 3 4 5 

No materials support Some materials support Rich and useful materials 

support 



d. Do the instructional materials provide information about the kinds of professional develop- 
ment experiences needed by teachers to implement the materials? 

1 2 3 4 5 

Little or no Partial information Rich and useful 

information provided provided information provided 



e. Do the materials provide guidance in how to link the materials with the district and state 
assessment frameworks and programs? 



1 


2 


3 


4 . . . . 




No guidance 




Some guidance 




Rich and useful guidance 
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f. Do the materials provide guidance and assistance for involving administrators, parents, and the 
community at large actively in supporting school mathematics? 

1 2 3 4 5 

No guidance Some guidance Rich and useful guidance 



g. Overall, are the materials usable by, realistic in expectations of, and supportive of teachers? 
1 2 3 4 5 

Teacher unfriendly Somewhat teacher friendly Teacher friendly 
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V. Major Strengths and Weaknesses 

a. In your opinion, what are the three major strengths of this curriculum? 



b. In your opinion, what are the three major weaknesses of this curriculum? 



* 16 9 
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VI. Overall Quality, Value, and Contribution 

a. In your opinion, what is the overall quality of these materials relative to: 

low high 



- turning students on to mathematics? 

- making students think? 

- quality of mathematics content? 

- quality of pedagogy? 

- quality of classroom assessments? 

- encouraging teachers to teach differently? 



1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



b. In your opinion, what is the overall quality of these instructional materials? 

1 . . 2 3 4 5 

Low Medium High 



c. To what extent would you encourage the dissemination, adoption, and implementation of this 
curriculum? 



1 2 , 

Not worthy of 
dissemination, adoption, 
nor implementation 



3 

OK to disseminate, 
adopt, and implement 
if revised 



OK to disseminate, 
adopt, and implement 
as is 
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Guidelines for Section II. 

The following is a brief outline of the National Council of Teachers of Mathematics Curriculum 
and Evaluation Standards for School Mathematics. It should be used to guide your responses to 
Section II. 



CONTENT STANDARD 


A. Mathematics as Problem Solving 


1. Use problem solving approaches to investigate and understand mathematical content 


2. Formulate problems from within and outside mathematics 


3. Develop and apply a variety of strategies to solve problems, with emphasis on multistep and 
nonroutine problems 


4. Verifv and interpret results with respect to the original problem situation 


5. Generalize solutions and strategies to new problem situations 


6. Acquire confidence in using mathematics meaningfully 


B. Mathematics as Communication 


1 . Model situations using oral, written, concrete, pictorial, graphical, and algebraic methods 


2. Reflect on and clarify their own thinking about mathematical ideas and situations 


3. Develop common understandings of mathematical ideas, including the role of definitions 


4. Use the skills of reading, listening, and viewing to interpret and evaluate mathematical ideas 


5. Discuss mathematical ideas and make conjectures and convincing arguments 


6. Appreciate the value of mathematical notation and its role in the development of mathematical 
ideas 


C. Mathematics as Reasoning 


1. Recognize and apply deductive and inductive reasoning 


2. Understand and apply reasoning processes, with special attention to spatial reasoning and 
reasoning with proportions and graphs 


3. Make and evaluate mathematical conjectures and arguments 


4. Validate their own thinking 


5. Appreciate the pervasive use and power of reasoning as a part of mathematics 


D. Mathematical Connections 


1 . See mathematics as an integrated whole 


2. Explore problems and describe results using graphical, numerical, physical, algebraic, and 
verbal mathematical models or representations 


3. Use a mathematical idea to further their understanding of other mathematical ideas 


4. Apply mathematical thinking and modeling to solve problems that arise in other disciplines, 
such as art, music, psychology, science, and business 


5. Value the role of mathematics in our culture and society 


E. Number and Number Relationships 


1. Understand, represent, and use numbers in a variety of equivalent forms (integer, fraction, 
decimal, percent, exponential, and scientific notation) in real-world and mathematical problem 
situations 


2. Develop number sense for whole numbers, fractions, decimals, integers, and rational numbers 


3. Understand and apply ratios, proportions, and percents in a wide variety of situations 


4. Investigate relationships among. fractions, decimals, and percents 


5. Represent numerical relationships in one- and two-dimensional graphs 
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CONTENT STANDARD 
(Continued) 


F. Number Systems and Number Theory 


1 . Understand and appreciate the need for numbers beyond the whole numbers 


2. Develop and use order relations for whole numbers, fractions, decimals, integers, and rational 
numbers 


3. Extend their understanding of whole number operations to fractions, decimals, integers, and 
rational numbers 


4. Understand how the basic arithmetic operations are related to one another 


5. Develop and apply number theory concepts (e.g., primes, factors, and multiples) in real-world 
and mathematical problem situations 


G. Computation and Estimation 


1. Compute with whole numbers, fractions, decimals, integers, and rational numbers 


2. Develop, analvze, and explain procedures for computation and techniques for estimation 


3. Develop, analvze, and explain methods for solving proportions 


4. Select and use an appropriate method for computing from among mental arithmetic, paper- 
and-pencil, calculator, and computer methods 


5. Use computation, estimation, and proportions to solve problems 


6. Use estimation to check the reasonableness of results 


H. Patterns and Functions 


1 . Describe, extend, analvze, and create a wide variety of patterns 


2. Describe and represent relationships with tables, graphs, and rules 


3. Analyze functional relationships to explain how a change in one quantity results in a change 
in another 


4. Use patterns and functions to represent and solve problems 


I. Algebra 


1. Understand the concepts of variable, expression, and equation 


2. Represent situations and number patterns with tables, graphs, verbal rules, and equations and 
explore the interrelationships of these representations 


3. Analyze tables and graphs to' identify properties and relationships 


4. Develop confidence in solving linear equations using concrete, informal, and formal methods 


5. Investigate inequalities and nonlinear equations informallv 


6. Apply algebraic methods to solve a variety of real-world and mathematical problems 


.1. Statistics 


1 . Systematically collect, organize, and describe data 


2. Construct, read, and interpret tables, charts, and graphs 


3. Make inferences and convincing arguments that are based on data analysis 


4. Evaluate arguments that are based on data analysis 


5. Develop an appreciation for statistical methods as powerful means for decision making 


K. Probability 


1 . Model situations by devising and carrying out experiments or simulations to determine 
probabilities 


2. Model situations bv constructing a sample space to determine probabilities 


3. Appreciate the power of using a probability model by comparing experimental results with 
mathematical expectations 
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CONTENT STANDARD 
(Continued) 


L. Geometry 


1 . Identify, describe, compare, and classify geometric figures 


2. Visualize and represent geometric figures with special attention to developing spatial 
sense 


3. Explore transformations of geometric figures 


4. Represent and solve problems using geometric models 


5. Understand and apply geometric properties and relationships 


6. Develop an appreciation of geometry as a means of describing the physical world 


M. Measurement 


1. Extend their understanding of the process of measurement 


2. Estimate, make, and use measurements to describe and compare phenomena 


3. Select appropriate units and tools to measure to the degree of accuracy required in a 
particular situations 


4. Understand the structure and use of systems of measurement 


5. Extend their understanding of the concepts of perimeter, area, volume, angle measure, 
capacity, and weight and mass 


6. Develop the concepts of rates and other derived and indirect measurements 


7. Develop formulas and procedures for determining measures to solve problems 
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Framework for Review 

Instructional Materials for High School Mathematics 1 



Title: 

Author(s): 

Publisher: 

Reviewed by: 

I. Descriptors 

a. Write a brief description of the components of the curriculum upon which this review is based 
(e.g., teachers guide, student books, hands-on materials, multimedia material). That is, what 
materials did you receive and include in your review? 



Copyright date: 
Date: 



b. Write a brief description of the purpose and broad goals of these materials. That is, what were 
the stated purposes and what were the actual purposes of the materials? 



c. What grade levels do the materials serve? 

9 10 11 12 



d. Are the instructional materials designed to 

provide a complete multi-year program for high school mathematics. 

provide a complete one-year course for high school mathematics. 

provide multiple modules or units that could be used to supplement other course materials 

for high school mathematics. 

provide a single module or collection of activities that could be used to supplement other 

course materials for high school mathematics. 

other (explain): 



' NOTE: This framework is adapted from an instrument developed by Inverness Research under contract to the 
National Science Foundation. The framework was refined as part of a panel review of NS F-supported materials for 
middle school science, which was limited to projects that provide at least a year-long course of study. 



Final Report on the Evaluation of the National Science Foundation’s Instructional Materials Development Program 



e. What are the major domains/topics of content covered by these materials? 
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II. Quality of the Mathematics 



Directions: For each item, circle the number corresponding with your response to the question. 

Write an explanation for your rating of each item below the item. 



a. Does the content in the instructional materials align well with all thirteen areas of the Curricu- 
lum Standards as described in the National Council of Teachers of Mathematics Curriculum and 
Evaluation Standards for School Mathematics '(NCTM)? 

(See attached guidelines) 



1 2 .... 

Omits substantial content 
included in NCTM 
and/or includes substantial content 
not recommended in NCTM 



3 

Some misalignment 
of content with 
recommendations 
in NCTM 



The curriculum 
aligns well with 
content recommendations 
in NCTM 



b. Are the mathematics concepts presented in the instructional materials accurate and correct? 
[Provide examples of major errors where they are evident. Attach extra page if necessary] 

1 2 3 4 5 

Substantial, major errors Mostly correct, with Mathematically accurate, 

some minor errors and correct 



c. Do the instructional materials adequately present the major concepts and adequately demon- 
strate and model the processes of mathematics? 

1 2 3 4 5 

Major concepts and Major concepts and Major concepts and 

processes not addressed processes somewhat addressed processes addressed well 
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d. Do the instructional materials accurately represent views of mathematical problem solving as de- 
scribed in the NCTM Curriculum and Evaluation Standards for School Mathematics? 

1 2 3 4 5 

Poor portrayal Mixed Rich and accurate 

of problem solving quality portrayal of 

problem solving 



e. Do the materials use technology as a tool for learning mathematics? 

1 2 3 4 5 

Little or no Some emphasis Rich and well 

use designed use 



f. Do the materials emphasize communication about mathematics through a variety of modalities? 



1 2 

Little or no 
emphasis, few 
modalities 



3 

Some emphasis, 
some modalities 



Rich and well 
designed emphasis, 
varied modalities 



g. Do the materials appropriately address mathematical reasoning? 

1 2 3 4 .5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 
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h. Do the materials appropriately address computation? 



1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



i. Do the materials appropriately address estimation? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



j. Do the materials appropriately address number systems? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



k. Do the materials appropriately address patterns? 

1 2 3 4 5 

Not appropriately addressed Somewhat appropriately addressed Appropriately addressed 



O 
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1. Do the materials appropriately address functions? 



1 2 


3 4 5 


Not appropriately addressed 


Somewhat appropriately addressed Appropriately addressed 



m. Do the materials appropriately address algebra? 



1 2 


3 4 5 


Not appropriately addressed 


Somewhat appropriately addressed Appropriately addressed 



n. Do the materials appropriately address geometry? 



1 2 


3 4 5 


Not appropriately addressed 


Somewhat appropriately addressed Appropriately addressed 



o. Do the materials appropriately address measurement? 



1 2 


3 4 5 


Not appropriately addressed 


Somewhat appropriately addressed Appropriately addressed 



p. Do the materials appropriately address statistics? 



1 2 


3 4 5 


Not appropriately addressed 


Somewhat appropriately addressed Appropriately addressed 
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q. Do the materials appropriately address probability? 



1 2 .... 

Not appropriately addressed 


Somewhat appropriately addressed Appropriately addressed 



r. Do the instructional materials provide sufficient activities for students to develop a good 
understanding of key mathematics concepts? 



1 2 .... 

Too few learning 
activities 


Activities provide Activities provide 

some opportunity many rich opportunities 

for students to learn to learn key mathematics 

some important concepts concepts 



s. Do the instructional materials provide sufficient opportunities for students to apply their 
understanding of the concepts (i.e., designing of solutions to problems or issues)? 



1 2 .... 

Very few 

application activities 


Some Very rich in 

application activities application activities 



t. Do the materials develop an appropriate breadth and depth of mathematics content? 



1 2 .... 

Too narrow 
or too broad 


Somewhat Good balance of 

balanced breadth and depth 



u. What is the overall quality of the mathematics presented in the instructional materials? 



1 2 .... 

Low 


Medium High 
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in. The Pedagogical Design 



a. Do the instructional materials provide a logical progression for developing conceptual under- 
standing in mathematics? 



1 

No logical 
progression 
of ideas 



2 



3 

Somewhat logical 
progression of ideas 



Logical progression 
of ideas that builds 
conceptual understanding 



b. Do the instructional materials provide students the opportunity to formulate, solve, and reflect 
critically on problems? 



1 


2 


3 


4 .... 


5 


No opportunity 




Some opportunity 




Rich and well 
designed opportunity 



c. To what extent are the mathematical concepts embedded in learner-appropriate contexts? 



1 2 . 

Very few or very 
contrived activities 
for students to do 
mathematical problem solving 



3 

Some good activities 
for students to do 
mathematical problem solving 



4 5 

Many rich and authentic 
opportunities for 
students to do 
mathematical 
problem solving 



d. How would you rate the overall developmental appropriateness of the instructional materials, 
given its intended audience of ALL students at the targeted level(s)? 

1 2 3 4 5 

Not developmental^ 
appropriate 



Somewhat developmentally Developmental ly 

appropriate appropriate 
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e. Do the materials reflect current (that is, within the last 5 years) knowledge about effective teaching and 
learning practices (e.g., active learning, inquiry, community of learners) based on research related to 
mathematics education? 



1 2 , 

Do not reflect 

current knowledge 

about teaching and learning 



3 4 

Somewhat reflective 
of current knowledge 
about teaching and learning 



. .5 

Reflect well 
current knowledge 
about teaching 
and learning 



f. Do the instructional materials provide students the opportunity to clarify, refine, and consolidate their 
ideas? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 



g. Do the instructional materials provide students with activities connecting mathematics with other 
subject areas? 

1 2 3 4 5 

No opportunity Some opportunity Rich and well 

designed opportunity 



h. Are the instructional materials likely to be interesting, engaging, and effective for girls and boys? 



1 2 

No sensitivity 
to gender issues 



3 



Some sensitivity 
to gender issues 



4 



. .5 

Sensitive to 
gender issues 
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i. Are the instructional materials likely to be interesting, engaging, and effective for underrepresented and 
underserved students (e.g., ethnic, urban, rural, with disabilities)? 

1 2 3 4 5 

No sensitivity Some sensitivity Sensitive to 

to underrepresented and to underrepresented and underrepresented 

and underserved students underserved students and underserved 

students 



j. Does assessment have explicit purposes connected to decisions to be made by teachers (e.g., prior 
knowledge, conceptual understanding, grades)? 

1 2 3 4 5 

Unclear purposes Somewhat clear Clear statement 

purposes of purposes 



k. Do assessments focus on the curriculum’s important content and skills? 

1 2 3 4 5 

Poor correspondence Fair correspondence Full correspondence 



1. Do the instructional materials include multiple kinds of assessments (e.g., performance, paper/pencil, 
portfolios, student interviews, embedded, projects)? 

1 2 3 4 5 

Little or no student Some variety of Complete 

assessment provided student assessment student assessment 

package 
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m. Are the assessment practices fair to all students? 



.4 



1 2 3 . . . 

Fair for a few Fair to most 



. .5 

Fair to all 



n. Do the instructional materials include adequate and appropriate uses of a variety of educa- 
tional technologies (e.g., calculators, video, computers, telecommunications)? 

1 2 3 4 5 

Little or no Some appropriate Many appropriate 

educational technology educational technology applications of 

included included educational 

technology 

included 



o. What is the overall quality of the pedagogical design of these instructional materials? 
1 2 3 4 5 

Low Medium High 
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IV. Implementation and System Support 

a. Will the teachers find the materials interesting and engaging? 

1 2 3 4 5 

Dry and boring Somewhat interesting Interesting and engaging 

and engaging 



b. Do the instructional materials include information and guidance to assist the teacher in imple- 
menting the lessons? 

1 2 3 4 5 

No teacher support Some teacher support Rich and useful teacher 

support 



c. Do the instructional materials provide information about the kind of resources and support 
system required to facilitate the district implementation of the required mathematics materials? 

1 2 3 4 5 

No materials support Some materials support Rich and useful materials 

support 



d. Do the instructional materials provide information about the kinds of professional develop- 
ment experiences needed by teachers to implement the materials? 



1 




3 


4 . . . . 


5 


Little or no 




Partial information 




Rich and useful 


information provided 
provided 




provided 




information 
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e. Do the materials provide guidance in how to link the materials with the district and state assessment 
frameworks and programs? 

1 2 3 4 5 

No guidance Some guidance Rich and useful guidance 



f. Do the materials provide guidance and assistance for involving administrators, parents, and 
the community at large actively in supporting school mathematics? 

1 2 3 4 5 

No guidance Some guidance Rich and useful guidance 



g. Overall, are the materials usable by, realistic in expectations of, and supportive of teachers? 
1 2 . 3 4 5 

Teacher unfriendly Somewhat teacher friendly Teacher friendly 
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V. Major Strengths and Weaknesses 

a. In your opinion, what are the three major strengths of this curriculum? 



b. In your opinion, what are the three major weaknesses of this curriculum? 
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VI. Overall Quality, Value, and Contribution 

a. In your opinion, what is the overall quality of these materials relative to: 

low high 



- turning students on to mathematics? 

- making students think? 

- quality of mathematics content? 

- quality of pedagogy? 

- quality of classroom assessments? 

- encouraging teachers to teach differently? 



1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



b. In your opinion, what is the overall quality of these instructional materials? 

1 2 3 4 5 

Low Medium High 



c. To what extent would you encourage the dissemination, adoption, and implementation of this 
curriculum? 

1 2 3 4 5 

Not worthy of OK to disseminate, OK to disseminate, 

dissemination, adoption, adopt, and implement adopt, and implement 

nor implementation if revised as is 
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Guidelines for Section II. 

The following is a brief outline of the National Council of Teachers of Mathematics Curriculum 
and Evaluation Standards for School Mathematics. It should be used to guide your responses to 
Section II. 



CONTENT STANDARD 


A. Mathematics as Problem Solving 


1. Use, with increasing confidence, problem solving approaches to investigate and understand 
mathematical content 


2. Apply integrated mathematical problem-solving strategies to solve problems from within and 
outside mathematics 


3. Recognize and formulate problems from situations within and outside mathematics 


4. AddIv the process of mathematical modeling to real-world problem situations 


B. Mathematics as Communication 


1. Reflect upon and clarify their thinking about mathematical ideas and relationships 


2. Formulate mathematical definitions and express generalizations discovered through 
investigations 


3. Express mathematical ideas orallv and in writing 


4. Read written presentations of mathematics with understanding 


5. Ask clarifying and extending questions related to mathematics thev have read or heard about 


6. Appreciate the economy, power, and elegance of mathematical notation and its role in the 
development of mathematical ideas 


C. Mathematics as Reasoning 


1 . Make and test conjectures 


2. Formulate counterexamples 


3. Follow logical arguments 


4. Judge the validity of arguments 


5. Construct simple valid arguments 


If college-intending. 


6. Construct proofs for mathematical assertions, including indirect proofs and proofs by 
mathematical induction 


D. Mathematical Connections 


1. Recognize equivalent representations of the same concept 


2. Relate procedures in one representation to procedures in an equivalent representation 


3. Use and value the connections among mathematical topics 


4. Use and value the connections between mathematics and other disciplines 


E. Algebra 


1. Represent situations that involve variable quantities with expressions, equations, inequalities, 
and matrices 


2. Use tables and graphs as tool to interpret expressions, equations, and inequalities 


3. Operate on expressions and matrices, and solve equations and inequalities 


4. Appreciate the power of mathematical abstraction and symbolism 


If college-intending. 


5. Use matrices to solve linear systems 


6. Demonstrate technical facility with algebraic transformations, including techniques based on 
the theory of equations 
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(Continued) 


F. Functions 


1. Model real-world phenomena with a variety of functions 


2. Represent and analyze relationships using tables, verbal rules, equations, and graphs 


3. Translate among tabular, symbolic, and graphical representations of functions 


4. Recognize that a variety of problem situations can be modeled bv the same type of function 


5. Analyze the effects of parameter changes on the graphs of functions 


If college-intending. 


6. Understand operations on, and the general properties and behavior of, classes of functions 


G. Geometry from a Synthetic Perspective 


1. Interpret and draw three-dimensional objects 


2. Represent problem situations with geometric models and applv properties of figures 


3. Classify figures in terms of congruence and similarity and applv these relationships 


4. Deduce properties of, and relationships between, figures from given assumptions 


If college-intending. 


5. Develop an understanding of an axiomatic system through investigating and comparing various 
geometries 


H. Geometry From an Algebraic Perspective 


1 . Translate between svnthetic and coordinate representations 


2. Deduce properties of figures using transformations and using coordinates 


3. Identify congruent and similar figures using transformations 


4. Analyze properties of Euclidean transformations and relate translations to vectors 


If college-intending. 


5. Deduce properties of figures using vectors 


6. Apply transformations, coordinates, and vectors in problem solving 


I. Trigonometry 


1. Apply trigonometry to problem situations involving triangles 


2. Explore periodic real-world phenomena using the sine and cosine functions 


If college-intending. 


3. Understand the connection between trigonometric and circular functions 


4. Use circular functions to model periodic real-world phenomena 


5. Applv general graphing techniques to trigonometric functions 


6. Solve trigonometric equations and verify trigonometric identities 


7. Understand the connections between trigonometric functions and polar coordinates, complex 
numbers, and series 


J. Statistics 


1. Construct and draw inferences from charts, tables, and graphs that summarize data from real-world 
situations 


2. Use curve fitting to predict from data 


3. Understand and applv measures of central tendency, variability, and correlation 


4. Understand sampling and recognize its role in statistical claims 


5. Design a statistical experiment to study a problem, conduct the experiment, and interpret and 
communicate the outcomes 


6. Analyze the effects of data transformations on measures of central tendency and variability 


If colle&e-intendine. 
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CONTENT STANDARD 
(Continued) 


K. Probability 


1. Use experimental or theoretical probability, as appropriate, to represent and solve problems 
involving uncertainty 


2. Use simulations to estimate probabilities 


3. Understand the concept of a random variable 


4. Create and interpret discrete probability distributions 


5. Describe, in general terms, the normal curve and use its properties to answer questions 
about sets of data that are assumed to be normally distributed 


If college-intending. 


6. Apply the concept of a random variable to generate and interpret probability distributions 
including binomial, uniform, normal, and chi sauare 


L. Discrete Mathematics 


1. Represent problem situations using discrete structures such as finite graphs, matrices, 
seauences, and recurrence relations 


2. Represent and analyze finite graphs using matrices 


3. Develop and analyze algorithms 


4. Solve enumeration and finite probability problems 


If college-intending. 


5. Represent and solve problems using linear programming and difference equations 


6. Investigate problem situations that arise in connection with computer validation and the 
application of algorithms 


M. Conceptual Underpinnings of Calculus 


1. Determine maximum and minimum points of a graph and interpret the results in problem 
situations 


2. Investigate limiting processes by examining infinite sequences and series and areas under 
curves 


If college-intending. 


3. Understand the conceptual foundations of limit, the area under a curve, the rate of change, 
and the slope of a tangent line, and their applications in other disciplines 


4. Analyze the graphs of polynomial, rational, radical, and transcendental functions 


N. Mathematical Structure 


1 . Compare and contrast the real number system and its various subsystems with regard to 
their structural characteristics 


2. Understand the logic of algebraic procedures 


3. Appreciate that seemingly different mathematical systems mav be essentially the same 


If college-intending. 


4. Develop the complex number system and demonstrate facility with its operation 


5. Prove elementary theorems within various mathematical structures, such as groups and 
fields 


6. Develop an understanding of the nature and purpose of axiomatic systems 



1 0 
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Classroom Observation Instrument 



IMD Product 

Classroom Observation Protocol 



Background Information 



IMD Product: 

Site: 

Subject/Grade Level: 
Observer: 



Date of Observation: 

Time of Observation: 

Start: End:, 



Section One: Contextual Background and Activities 

In this section, please fill in the circles that best describe the class. For each item* be sure to fill in all 
responses that apply. 



L Classroom Demographics 



A . What is the total number of students in 
the class at the time of the 
observation? 

O 1 5 or fewer 
O 16-20 
O 21-25 
O 26-30 
O 3 1 or more 

C. Indicate the gender and racc/cnthnicity 
of the teacher ? 



B. What is the approximate percentage of 
white (not Hispanic origin) students in 
the class? 

O 0-10 percent 
O 1 1-25 percent 
O 26-50 percent 
O 51-75 percent 
O 76- 1 00 percent 

D. Indicate the gender and racc/cnthnicity 
of the teacher's aide ? 



D Male O Female 

O African-American (not Hispanic origin) 
O American Indian or Alaskan Native 
P Asian or Pacific Islander 
O Hispanic 

O White (not Hispanic origin) 

O Other 



O Male O Female 

O African-American (not Hispanic origin) 
O American Indian or Alaskan Native 
O Asian or Pacific Islander 
O Hispanic 

O White (not Hispanic origin) 

O Other 

BUST COPY AVAILABLE 
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II. Classroom Context 

A. Rate the adequacy of the physical environment. 
I. Classroom resources: 



0 O 

1 2 
Sparsely equipped 



2. Classroom space: 

0 

1 

Crowded 



O 

2 



3. Room arrangement: 

0 O 

1 2 
Inhibited interactions 

among students 



O 

3 



O 

3 



O 



o o 

4 5 

Rich in 
resources 



O O 

4 5 

Adequate 

space 



O O 

4 5 

Facilitated 
interactions among 
students 



B. In a few sentences, describe the lesson you observed. Include where this lesson fits in the overall 
unit of study, if you know (his. 



III. Lesson Focus 

A. Indicate the major 0 content area(s) of this lesson or activity. 



o 


Numeration and number theory 


o 


Data collection and analysis 


o 


Computation 


o 


Probability 


o 


Estimation 


o 


Statistics (e.g., hypothesis tests, curve 


o 


Measurement 




fitting, and regression) 


o 


Pre-algebra 


o 


Topics from discrete mathematics (c.g.. 


o 


Algebra 




combinatorics, graph theory, recursion) 


o 


Pattcrns-and relationships 


o 


Mathematical structures (e,g., vector 


o 


Geometry and spatial sense 




spaces; groups, rings, fields) 


0 


Functions (including trigonometric 
functions) and prc-calculus concepts 


o 


Calculus 




"Major** means, was used, or addressed for a substantial portion of the lesson; if you were describing the lesson to 



someone, this feature would help characterize it. 
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O Life science 
O Physical science 
O Earth and space science 
O Environmental science 

O Engineering and design principles 
O Technology (calculators, computers) in 
support of science/mathematics 



O History of science/mathematics 
O Other disciplines (please specify:) 



IV. Classroom Instruction 



A. Indicate the major instructional resource(s) used in this lesson. Please specify the names of 
any published teacher guides, textbooks, or kits used in this lesson. 



D Print Materials: 

O Published teacher guide/manua! for 
hands-on unit: 



O Published textbook(s): 



O Teacher-created print materials 
O Other published materials (e.g.. trade 
books, magazines) 

O Hands*on/Manipulative Materials/ 

Models: 

0 Tools and instruments 
O Objects, specimens, or models 
O Commercially-produced manipulativcs 
O Comincrcialiy-produeed kits: 

O Other hands-on/laboratory supplies 



O Outdoor Resources: 

O Garden 
O Nature trail 
O Other outdoor area 

O TcchnoIogy/Audio-visual Resources: 

O Computers 
O Calculators 

O Videotapc/ftlm/filmstrip/TV program 
O Multimedia 
O Telecommunications 

O Other Resources: 

O Chalkboard 
O Overhead Projector 
O Chans 
O Maps 



1) . Indicate the major way(s) in which student activities were structured. 

O As a whole group 
0 As small groups 
0 As pairs 
O As individuals 



BEST 



AVAILABLE 



• “Major" means, was used, or addressed for a substantial portion of the lesson; if you were describing the lesson to 



someone, this feature would help characterizes. 
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C. Indicate the major* way(s) in which students engaged in class activities. 

O Entire class was engaged in the same activities at the same time. 

O Croups of students were engaged in different activities at the same time (e.g.. centers). 



V. Use of Material 

A. Indicate the major activities of teachers and students in this lesson. When choosing an 
"umbrella" category, be sure to indicate subcategories that apply as well. (For example, if you mark 
"formal presentations by teacher.” indicate what kind). 



3 Formal Presentations by Teacher: 

O Disciplinary content/process 
information 

O Demonstration of a principle or 
phenomenon 

O Procedural instructions 
O Other (please speci fy) 



O Students Presenting and/or Defending Work 
Orally 

3 Guest Speaker/”Expert" Serving as a 
Resource 

D Discussions/Seminars: 

O Whole group led by teacher 
O Whole group led by siudent(s) 

O Small groups/pairs 

D Students Engaged in Hands- 

on/In vestigative/Rescarch/FicId Activities: 

O Followed prescribed steps in a 
science/mathematics activity or 
investigation 

O Designed or implemented their own 
investigation in science/mathematics 
O Worked on an extended 

investigation/projcct (a week or more in 
duration) 

O Recorded, represented, and/or analyzed 
dan 

O Interpreted data to draw conclusions 
O Worked on a model or simulation 
O Designed objects within constraints 
(e.g.. egg drop, toothpick bridges, 
aluminum boats) 



O Participated in fteldwork 
O Engaged in role play or debate 
O Played a game to build or review 
knowledge/skills 

O Students Engaged in Problem-Solving 
Activities: 

O Practiced routine 

computations/algorithums 
O Determined if a problem was well 
defined 

O Reflected on examples of problems and 
their solutions 

O Recognized patterns, cycles, or trends 
O Worked on solving a real- world or 
abstract problem 

O Applied scicntific/mathematical 

principles or strategies in solving new 
problems 

O Formulated conjectures to generalize 
problems 

O Students Focused on Proof and Evidence: 

O Reflected on methods of proof in 
scicncc/mathematics 

O Evaluated the validity of arguments or 
claims 

O Tested a hypothesis or conjecture 
O Developed a formal argument or proof 

O Students Engaged in 

Rcading/RcflcctionAVritlen Communication: 

O Read (or listened to a story) about 
science/mathematics 

O Answered tcxtbook/workshect questions 
O Reflected on readings, activities, or 
problems individually or in groups 



the lesson to 



* “Major’* means, was used, or addressed for a substantial portion ofthe lesson: if you were describing 
someone, this feature would help characterize it. 
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o 


Wrote a description of a plan, procedure. 








or problem-solving process 


O Students participated in assessment: 


o 


Wrote a reflections in a notebook or 
journal 


o 


Homework/worksheet review 


o 


Prepared a written product (e.g., report, 


o 


Informal assessment (e.g.. questioning 




story, poem) 


o 


for understanding) 

Short-answer tests (e.g.. multiple 


Students and/or Teacher Used 




choice, true/false, fiil-in-the-blank) 


Technology/Audio-visual Resources: 


o 


Tests requiring open-ended responses 








(e.g.. explanations, descriptions, or 


o 


To develop conceptual understanding 




justifications of solutions) 


o 


To learn or practice a skill 


o 


Performance-based assessment 


o 


To collect data (e.g.. probeware) 


o 


Embedded assessment (using class 


o 


As an analytic tool (e.g.. spreadsheets or 




activities/projects for assessment 




data analysis) 




purposes) 


o 


As a presentation tool 


o 


Portfolios 


o 


For word processing 






o 


As a communication tool (e.g.. 
e-mail. Internet. Web) 







D Other activities (please specify): 



E. Comments 

Please provide any additional information you consider necessary to capture the activities or context of this lesson. 
Include comments on any feature of the class that is so salient that you need to get It “on the table** right away to 
help explain your ratings; for example, the class was interrupted by a fire drill, the kids were excited about an 
upcoming school event, or the teacher’s tone was so warm (or so hostile) that it was an overwhelmingly important 
feature of the lesson. 
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Section Two: Ratings 

In Section One of this form, you documented what occurred in the lesson. In this section, you are asked to rate each 
of a number of key indicators from l (not at all) to 5 (to a great extent) in four different categories by circling the 
appropriate response. You may list any additional indicators you consider important in capturing the essence of this 
session and rate these as well. Use your “Ratings of Key Indicators" (Part A) to inform your ' Synthesis Ratings" 
(Part B) and indicate in "Supporting Evidence for Synthesis Ratings" (Part C) what factors were most influential in 
determining your synthesis ratings. Note that any one lesson is not likely to provide evidence for every single 
indictor; use 6, "Don’t Know" when there is not enough evidence for you to make a judgment. Use 7. "N/A" (Not 
Applicable) when you consider the indicator inappropriate given the purpose and context of the session. Section 
Two concludes with ratings of the likely impact of instruction, and a capsule description of the lesson. 



f« Implementation of Key Strategies Embedded in the Curriculum Product 



A. Ratings of Key Indicators 



Not 

at all 

1 . The implementation of instructional strategies was 

consistent with investigative science/ I 2 3 

mathematics. 

2. The teacher appeared confident in his/her ability to use l 2 3 

the materials. 

3. The teacher’s classroom management stylc/sirategies 

enhanced the quality of the lesson. i 2 3 

4. The pace of the lesson was appropriate for the 

developmental levels/needs of the students and the 12 3 

purposes of the lesson, as embedded in the materials. 

5. The teacher was cognizant of prior knowledge of students 1 2 3 

6. The teacher’s questioning strategics were likely to 

enhance the development of student conceptual I 2 3 

understanding (e.g.. emphasized higher order questions, 
appropriately used "wait time"). 

7. Teacher used assessment strategics appropriate to the l 2 3 

lesson and materials. 

S. I 2 3 



4 



To a 



great Don’t 
extent Know 



5 



6 



N/A 



7 



4 5 



6 7 



4 

4 



5 



6 



5 



6 



7 

*7 



4 5 



6 7 



4 5 



6 7 



4 5 



6 7 



4 5 



6 7 



B. Synthesis Ra tine 



1 


2 


3 


4 


5 


Implementation of 
iessun not at all 
reflective of best 
practice 








Implementation of 
lesson extremely 
reflective of best 
Practice 



C. Supporting evidence for Synthesis Rating 
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II. Content 



A. Ratings of Key Indicators 





Not 
at all 








To a 
great 
extent 


Don’t 

Know 


N/A 


L 


The content was appropriate for the purposes of the 1 

lesson and developmental level of the class. 


2 


3 


4 


5 


6 


7 


2. 


The science/mathematics content was significant and 1 

worthwhile. 


2 


3 


4 


5 


6 


7 


3. 


Teacher-presented information was accurate. I 


2 


3 


4 


5 


6 


7 


4. 


The teacher was familiar with the materials and their I 

intentions. 


2 


3 


4 


5 


6 


7 


5. 

6. 


The content was relevant for the needs/interests of most 1 

students. 

Science/mathematics was portrayed as a dynamic body of 


2 


3 


4 


5 


6 


7 


7. 


knowledge that involves conjecture, investigation, 1 

analysis, and proof/justification. 

Appropriate connections were made to other areas of 


2 


3 


4 


5 


6 


7 


S. 


science/mathematics, to other disciplines, and/or to real- 1 

world contexts. 

The degree of closure or resolution of conceptual 


2 


3 


4 


5 


6 


7 




understanding was appropriate for the developmental 1 

levcls/necds of the students and the purposes of the 
lesson. 


2 


3 


4 


5 


6 


7 


9. 


The materials were used in the lesson. 1 


2 


3 


4 


5 


6 


7 


10. 


1 


2 


3 


4 


5 


6 


7 



B. Synthesis Rating 



i 


2 


3 


4 


5 


Content of lesson 
not at nil reflective 
of current standards 
for science/ 
mathematics 
education 








Content of lesson 
extremely reflective 
of current standards 
for science/ 
mathematics 
education 



C. Supporting evidence for Synthesis Rating 
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III. Classroom Culture/Equity 



A. Ratings of Key Indicators 















To a 










Not 








great 


Don’t 








at all 








extent 


Know 


N/A 


I. 


Active participation of all was encouraged and valued. 


1 


2 


3 


4 


5 


6 


7 


2. 


There was a climate of respect for students’ ideas, 
questions, and contributions. 


l 


2 


3 


4 


5 


6 


7 


3. 


Interactions reflected collaborative working relationships 
among students (e.g.. students worked together, talked 
with each other about the lesson). 


l 


2 


3 


4 


5 


6 


7 


4 . 


Interactions reflected collaborative working relationships 
between teacher and students. 


1 


2 


3 


4 


5 


6 


7 


5 . 


The teacher’s language and behavior demonstrated 
sensitivity to issues of gender, race/cthnicity, special 
needs, limited English proficiency, culture, and socio- 
economic status. 


1 


2 


3 


4 


5 


6 


7 


6. 


Opportunities were taken to recognize and challenge 
stereotypes and biases that became evident during the 
lesson. 


1 


2 


3 


4 


5 


6 


7 


7. 


Students were intellectually engaged with important ideas 
relevant to the focus of the lesson. 


1 


2 


3 


4 


5 


6 


7 


S. 


Students were encouraged to generate ideas, questions, 
conjectures, or propositions. 


I 


2 


3 


4 


5 


6 


7 


9. 


Intellectual rigor, constructive criticism, and the 
challenging of ideas were valued. 


1 


2 


3 


4 


5 


6 


7 


10. 




1 


2 


3 


4 


5 


6 


7 



13. Svnthcsis Ratine 



I 


2 


3 


4 


5 


The classroom 
culture interferes 
with student learning 








The classroom 
culture facilitates the 
learning of all 
students 



C. Supporting evidence for Synthesis Rating 
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IV, Overall Ratings of the Lesson 

A, Likely Impact of Instruction on Students’ Understanding of Science/Mathematics 
While the impact of a single lesson may well be limited in scope, it is important to judge whether it is helping 
move students in the desired direction. For this series of ratings, consider all available information (i.e. t your 
previous ratings of design, implementation, content, and culture/equity and the pre-and post-observation interviews 
with the teacher) as you assess the probable impact of this lesson. Fee free to elaborate on ratings with comments 
in the space provided. 



Circle the response that best describes your overall assessment of the likely effect of this lesson in each of the 
following areas. 





Negative 

effect 




Neutral 

effect 




Positive 

effect 


Don’t 

Know 


N/A 


1 . Students* understanding of science as a dynamic 
body of knowledge generated and enriched by 
investigation. 


I 


2 


3 


4 


5 


6 


7 


2. Students* understanding of mathematics by the 
use of multiple approaches. 

3. Students* understanding of important 


1 


2 


3 


4 


5 


6 


7 


science/mathematics concepts. 


1 


2 


3 


4 


5 


6 


7 


4. Students* capacity to carry out their own 
inquiries, 

5. Students* ability to apply or generalize skills 


I 


2 


3 


4 


5 


6 


7 


and concepts to other areas of 
science/mathematics, other disciplines, and/or 
real-life situations. 

6. Students* self-confidence in doing 


l 


2 


3 


4 


5 


6 


7 


science/mathematics. 

7. Students* interest in and/or appreciation for the 


i 


2 


3 


4 


5 


6 


7 


discipline 


1 


2 


3 


4 


5 


6 


7 



Comments (optional): 
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B. Capsule Description of the Quality of the Lesson 

In this final rating of the lesson, consider all available information about the lesson, its context and purpose, and 
your own judgment of the relative importance of the ratings you have made. Select the capsule description that best 
characterizes the lesson you observed. Keep in mind that this rating is not intended to be an average of all the 
previous ratings, but should encapsulate your overall assessment of the quality and likely impact of the lesson. Feel 
free to comment on or modify the capsule description in the space provided for comments. 

O Level I: Ineffective Instruction 

There is little or no evidence of student thinking or engagement with important ideas of 
science/mathematics. Instruction is unlikely to enhance students* understanding of the discipline or to 
develop their capacity to successfully “do** science/mathematics. Lesson was characterized by either (mark 
one below): 

O Passive Learning 

Instruction is pedantic and uninspiring. Students are passive recipients of information, from the teacher 
or textbook: material is presented in a way that is inaccessible to many students. 

O Activity for Activity’s Sake 

Students are involved in hands-on activities or other individual or group work, but it appears to be 
activity for activity’s sake. Lesson lacks a clear sense of purpose and/or a clear link to conceptual 
development. 

3 Level 2: Elements of Effective Instruction 

Instruction contains some elements of effective practice, but there arc substantial problems in the design, 
implementation, content, and/or appropriateness for many students in the class. For example, the content 
may lack importance and/or appropriateness; instruction may not successfully address the difficulties that 
many students are experiencing, etc. Overall, the lesson is quite limited in its likelihood to enhance 
students* understanding of the discipline or to develop their capacity to successfully “do** 
science/mathematics. 

O Level 3: Beginning Stages of Effective Instruction 

Instruction is purposeful and characterized by quite a few elements of effective practice. Students are, at 
times, engaged in meaningful work, but there are some weaknesses in the design, implementation, or 
content of instruction. For example, the teacher may short-circuit a planned exploration by telling students 
what they "should have found,** instruction may not adequately address the needs of a number of students, or 
the classroom culture may limit the accessibility or effectiveness of the lesson. Overall, the lesson is 
somewhat limited in its likelihood to enhance students* understanding of the discipline or to develop their 
capacity to successfully "do*’ scicncc/mathematics. 

3 Level 4: Accomplished. Effective Instruction 

Instruction is purposeful and engaging for most students. Students actively participate in meaningful work 
(c.g.. investigations, teacher presentations, discussions with each other or teacher, reading). The lesson is 
well-designed and the teacher implements it well, but adaptation of content or pedagogy in response to 
student needs and interests is limited. Instruction is quite likely to enhance most students’ understanding of 
the discipline and to develop their capacity to successfully **do“ science/mathematics. 

3 Level 5: Exemplary Instruction 

Instruction is purposeful and all students arc highly engaged most or all of the time in meaningful work 
(c.g.. investigations, teacher presentations, discussions with each other or teacher, reading). The lesson is 
wdl-dcsigncd and artfully implemented, with flexibility and responsiveness to student needs and interests. 
Instruction is highly likely to enhance most students' understanding of the discipline and to develop their 
capacity to successfully "do‘* science/mathematics. 

Comments (optional): 
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The National Science Foundation (NSF) funds research and education in most fields of science 
and engineering. Grantees are wholly responsible for conducting their project activities and 
preparing the results for publication. Thus, the Foundation does not assume responsibility for 
such findings or their interpretation. 

NSF welcomes proposals from all qualified scientists, engineers and educators. The Founda- 
tion strongly encourages women, minorities, and persons with disabilities to compete fully in its 
programs. In accordance with federal statutes, regulations, and NSF policies, no person on 
grounds of race, color, age, sex, national origin, or disability shall be excluded from participa- 
tion in, be denied the benefits of, or be subjected to discrimination under any program or 
activity receiving financial assistance from NSF (unless otherwise specified in the eligibility 
requirements for a particular program). 

Facilitation Awards for Scientists and Engineers with Disabilities (FASED) provide funding for 
special assistance or equipment to enable persons with disabilities (investigators and other 
staff, including student research assistants) to work on NSF-supported projects. See the 
program announcement or contact the program coordinator at (703) 306-1636. 

The National Science Foundation has Telephonic Device for the Deaf (TDD) and Federal 
Information Relay Service (FIRS) capabilities that enable individuals with hearing impairments 
to communicate with the Foundation regarding NSF programs, employment, or general infor- 
mation. TDD may be accessed at (703) 306-0090 or through FIRS on 1-800-877-8339. 

The National Science Foundation is committed to making all of the information we publish easy 
to understand. If you have a suggestion about how to improve the clarity of this document or 
other NSF-published materials, please contact us at plainlanguage@nsf.gov. 
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